Novartis Experiments failing when calling calc_pdx_metrics.py on AWS

[jjacobson95]

This ran successfully last time around, but now has failed twice in a row when running on the AWS EC2 instance. Currently looking into this.

Error Message:

Traceback (most recent call last):
  File "/app/calc_pdx_metrics.py", line 418, in <module>
    main()
  File "/app/calc_pdx_metrics.py", line 323, in main
    singles, combos = get_drug_stats(tab)
  File "/app/calc_pdx_metrics.py", line 390, in get_drug_stats
    lmm_res = lmm(comb.time, comb.volume, comb.treatment, d)
  File "/app/calc_pdx_metrics.py", line 289, in lmm
    fit = model.fit()
  File "/usr/local/lib/python3.10/site-packages/statsmodels/regression/mixed_linear_model.py", line 2257, in fit
    pcov = np.linalg.inv(-hess)
  File "<__array_function__ internals>", line 180, in inv
  File "/usr/local/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 552, in inv
    ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
  File "/usr/local/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 89, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
numpy.linalg.LinAlgError: Singular matrix

[sgosline]

Can you try to figure out which line of the table (tab) this fails on? It's possible that the error is happening locally but somehow being ignored on your own architecture.

[jjacobson95]

Its failing on this line:

fit = model.fit()

Added debug lines:
Local version:

[DBG][LMM] DATA:       model_id  volume  time exp_type  log_volume
10042    model   218.3     0  control    5.385870
10043    model   273.0     3  control    5.609472
10044    model   341.6     7  control    5.833640
10045    model   431.7    11  control    6.067731
10046    model   485.9    13  control    6.186003
10047    model   662.1    17  control    6.495417
10048    model   859.5    21  control    6.756351
10049    model  1146.8    24  control    7.044731
10050    model  1352.0    28  control    7.209340
9982     model   233.3     0   ljm716    5.452325
9983     model   241.5     3   ljm716    5.486869
9984     model   235.8     7   ljm716    5.462984
9985     model   234.7    10   ljm716    5.458308
9986     model   232.7    14   ljm716    5.449750
9987     model   268.4    17   ljm716    5.592478
9988     model   365.2    21   ljm716    5.900445
9989     model   414.7    25   ljm716    6.027555
9990     model   487.4    27   ljm716    6.189085
9991     model   506.9    31   ljm716    6.228314
9992     model   557.8    35   ljm716    6.324000
9993     model   651.6    38   ljm716    6.479431
9994     model   726.8    42   ljm716    6.588651
9995     model   755.5    44   ljm716    6.627380
9996     model  1002.9    50   ljm716    6.910651
[DBG][LMM] fit: Intercept                  5.373519
exp_type[T.ljm716]        -0.128355
time                       0.066346
time:exp_type[T.ljm716]   -0.034750
Group Var                  1.000000
dtype: float64
[DBG][LMM] i_coef_value: -0.03475030459910886

AWS Version:

[DBG][LMM] DATA:       model_id  volume  time exp_type  log_volume
10042    model   218.3     0  control    5.385870
10043    model   273.0     3  control    5.609472
10044    model   341.6     7  control    5.833640
10045    model   431.7    11  control    6.067731
10046    model   485.9    13  control    6.186003
10047    model   662.1    17  control    6.495417
10048    model   859.5    21  control    6.756351
10049    model  1146.8    24  control    7.044731
10050    model  1352.0    28  control    7.209340
9982     model   233.3     0   ljm716    5.452325
9983     model   241.5     3   ljm716    5.486869
9984     model   235.8     7   ljm716    5.462984
9985     model   234.7    10   ljm716    5.458308
9986     model   232.7    14   ljm716    5.449750
9987     model   268.4    17   ljm716    5.592478
9988     model   365.2    21   ljm716    5.900445
9989     model   414.7    25   ljm716    6.027555
9990     model   487.4    27   ljm716    6.189085
9991     model   506.9    31   ljm716    6.228314
9992     model   557.8    35   ljm716    6.324000
9993     model   651.6    38   ljm716    6.479431
9994     model   726.8    42   ljm716    6.588651
9995     model   755.5    44   ljm716    6.627380
9996     model  1002.9    50   ljm716    6.910651
Traceback (most recent call last):
  File "/app/calc_pdx_metrics.py", line 473, in <module>
    main()
  File "/app/calc_pdx_metrics.py", line 362, in main
    singles, combos = get_drug_stats(tab)
  File "/app/calc_pdx_metrics.py", line 445, in get_drug_stats
    lmm_res = lmm(comb.time, comb.volume, comb.treatment, d)
  File "/app/calc_pdx_metrics.py", line 324, in lmm
    fit = model.fit()
  File "/usr/local/lib/python3.10/site-packages/statsmodels/regression/mixed_linear_model.py", line 2257, in fit
    pcov = np.linalg.inv(-hess)
  File "<__array_function__ internals>", line 180, in inv
  File "/usr/local/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 552, in inv
    ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
  File "/usr/local/lib/python3.10/site-packages/numpy/linalg/linalg.py", line 89, in _raise_linalgerror_singular
    raise LinAlgError("Singular matrix")
numpy.linalg.LinAlgError: Singular matrix