Table 2 Comparison of the allele-frequency estimates and called genotypes by different methods with low depths of coverage
MethodqfFormula (mean ± 2 SEM)Correct-Call Rate among Individuals (mean ± 2 SEM)Correct-Call Rate among Called Genotypes (mean ± 2 SEM)
Proposed0.10 (HWE)0.10 ± 0.000500.84 ± 0.000640.91 ± 0.00049
GATK0.10 (HWE)0.06 ± 0.000420.80 ± 0.000730.88 ± 0.00057
Samtools0.10 (HWE)0.08 ± 0.000390.90 ± 0.000450.90 ± 0.00045
ANGSD0.10 (HWE)0.10 ± 0.000500.90 ± 0.000450.90 ± 0.00045
Proposed0.1Minimized0.10 ± 0.000500.87 ± 0.000680.95 ± 0.00049
GATK0.1Minimized0.06 ± 0.000410.82 ± 0.000770.91 ± 0.00061
Samtools0.1Minimized0.08 ± 0.000390.94 ± 0.000470.94 ± 0.00047
ANGSD0.1Minimized0.11 ± 0.000500.94 ± 0.000460.94 ± 0.00046
Proposed0.1Maximized0.10 ± 0.000630.95 ± 0.000461.00 ± 0.00015
GATK0.1Maximized0.09 ± 0.000580.93 ± 0.000511.00 ± 0.00006
Samtools0.1Maximized0.06 ± 0.000460.91 ± 0.000470.91 ± 0.00047
ANGSD0.1Maximized0.08 ± 0.000570.91 ± 0.000470.91 ± 0.00047
Proposed0.30 (HWE)0.30 ± 0.000770.77 ± 0.000810.88 ± 0.00080
GATK0.30 (HWE)0.22 ± 0.000770.68 ± 0.000940.79 ± 0.00088
Samtools0.30 (HWE)0.25 ± 0.000960.84 ± 0.000740.84 ± 0.00074
ANGSD0.30 (HWE)0.30 ± 0.000760.84 ± 0.000740.84 ± 0.00074
Proposed0.3Minimized0.30 ± 0.000630.74 ± 0.000870.87 ± 0.00075
GATK0.3Minimized0.19 ± 0.000670.58 ± 0.000980.70 ± 0.00101
Samtools0.3Minimized0.26 ± 0.000850.83 ± 0.000780.83 ± 0.00078
ANGSD0.3Minimized0.31 ± 0.000640.83 ± 0.000780.83 ± 0.00078
Proposed0.3Maximized0.30 ± 0.000940.94 ± 0.000471.00 ± 0.00017
GATK0.3Maximized0.26 ± 0.000930.90 ± 0.000611.00 ± 0.00011
Samtools0.3Maximized0.24 ± 0.001180.87 ± 0.000670.87 ± 0.00067
ANGSD0.3Maximized0.28 ± 0.001020.87 ± 0.000670.87 ± 0.00067
  • q, Formula and f are the minor-allele frequency, its estimate, and inbreeding coefficient, respectively. Formula by the proposed method and ANGSD are directly estimated from sequence read data by the genotype-frequency estimator (Maruki and Lynch 2015) and Kim et al.’s (2011) method, respectively. Called genotypes by the proposed method are by the Bayesian genotype caller. The correct-call rate among individuals is a fraction of individuals with correctly called genotypes among N = 100 individuals, where missing genotype calls are considered incorrect. On the other hand, the correct-call rate among called genotypes is calculated only among individuals with called genotypes. Mean depth of coverage µ = 3, error rate ε = 0.01. Results are based on a total of 10,000 simulation replications for each parameter set. HWE, Hardy–Weinberg equilibrium.