CrystFEL issueshttps://gitlab.desy.de/thomas.white/crystfel/-/issues2024-01-02T15:27:53+01:00https://gitlab.desy.de/thomas.white/crystfel/-/issues/24Add a way to move panels manually in GUI2024-01-02T15:27:53+01:00Thomas WhiteAdd a way to move panels manually in GUIWe need a replacement for the "calibration mode" in old hdfsee.We need a replacement for the "calibration mode" in old hdfsee.CrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/92Write a single Millepede file2024-03-01T13:39:41+01:00Thomas WhiteWrite a single Millepede fileAt the moment, we create one Millepede file per indexamajig worker process. That gets multiplied by the number of indexamajig jobs running across the cluster. This results in **thousands** of files, which is not nice for the filesystem...At the moment, we create one Millepede file per indexamajig worker process. That gets multiplied by the number of indexamajig jobs running across the cluster. This results in **thousands** of files, which is not nice for the filesystem or for humans to keep track of. It would be better to combine the Millepede data into a single file for each run of indexamajig. It could be done in a similar way to how the streams get combined from each worker process.CrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/91Per-panel peak lists2024-02-20T11:04:28+01:00Thomas WhitePer-panel peak listsThis is a proposal for an extension to the geometry file to enable a separate peak list for each panel. This would allow for peak lists without "slabby" data. The new option `panel_peak_list` would work almost exactly like the old `pea...This is a proposal for an extension to the geometry file to enable a separate peak list for each panel. This would allow for peak lists without "slabby" data. The new option `panel_peak_list` would work almost exactly like the old `peak_list`, but would interpret the coordinates as relative to the panel. It would be mutually exclusive with old `peak_list`, which would assume the same "slabby" interpretation as before. `panel_peak_list` would be a per-panel attribute, whereas `peak_list` is top-level only.
Simple case with one image per file, one array per panel:
```
panelA/data = /location/panelA/imagedata
panelA/panel_peak_list = /location/panelA/peaks
panelB/data = /location/panelB/imagedata
panelB/panel_peak_list = /location/panelB/peaks
```
Or, one image per file, 3D array with all panels together:
```
data = /location/data3Darray
dim1 = ss
dim2 = fs
panelA/dim0 = 0
panelA/panel_peak_list = /location/peaklists/panelA
panelB/dim0 = 1
panelB/panel_peak_list = /location/peaklists/panelB
```
Note: no support for a weird multi-dimensional peak list somehow matching the image array. However, the following would work: multiple images per file, one array per panel with associated peak list:
```
panelA/data = /location/images/%/panelA/imagedata
panelA/panel_peak_list = /location/images/%/panelA/peaks
panelB/data = /location/images/%/panelB/imagedata
panelB/panel_peak_list = /location/images/%/panelB/peaks
```
The `cxi` peak list format implements a "weird multi-dimensional peak list", but it makes assumptions and only works for exactly one dimension of image number.CrystFEL 0.12.0Thomas WhiteThomas Whitehttps://gitlab.desy.de/thomas.white/crystfel/-/issues/89Partialator deltaCChalf seems to result in twinning2024-01-02T15:28:47+01:00Thomas WhitePartialator deltaCChalf seems to result in twinningI've had a report of apparently twinned data from partialator when using large numbers of iterations. The fact that it only happens with larger numbers of iterations already narrows it down a bit: with no post-refinement, it should be a...I've had a report of apparently twinned data from partialator when using large numbers of iterations. The fact that it only happens with larger numbers of iterations already narrows it down a bit: with no post-refinement, it should be almost completely stable regardless of the number of iterations. On testing, adding `--no-deltacchalf` removed the apparent instability.
I don't know if this is expected for the algorithm, or due to a bug. In any case, the problem has been there for a long time...CrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/86Indexamajig should exec after fork2023-10-04T15:05:00+02:00Thomas WhiteIndexamajig should exec after forkIndexamajig does something naughty in creating its worker processes: it calls `fork` and carries on without calling `exec`. Technically, this is not allowed - only async-safe calls are allowed between `fork` and `exec`. So far, we get ...Indexamajig does something naughty in creating its worker processes: it calls `fork` and carries on without calling `exec`. Technically, this is not allowed - only async-safe calls are allowed between `fork` and `exec`. So far, we get away with this because none of the libraries create any threads or locks that might conflict. However, this might change at any time.
Instead, indexamajig should replace the entire process image with `exec`, and carry across only the necessary resources (file descriptors and shared memory). Or, use a different API such as `posix_spawn`.
Changing this will need a big effort (though, not *that* big). Or, just leave it because [it's good enough for Python](https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods)?https://gitlab.desy.de/thomas.white/crystfel/-/issues/80Faster reflection prediction2024-01-02T15:30:38+01:00Thomas WhiteFaster reflection predictionCrystFEL is currently using a naive brute force algorithm, looping over all reflections to determine which ones are excited. There are at least two better algorithms available:
1. Region growing algorithm from Wolfgang Brehm. See [thi...CrystFEL is currently using a naive brute force algorithm, looping over all reflections to determine which ones are excited. There are at least two better algorithms available:
1. Region growing algorithm from Wolfgang Brehm. See [this paper](https://doi.org/10.1107/S2053273323000682) (appendix B). Preliminary work on branch [region-growing](https://gitlab.desy.de/thomas.white/crystfel/-/commits/region-growing).
2. Quadratic limits algorithm from G. N. Reeke. Mentioned in [this paper](http://scripts.iucr.org/cgi-bin/paper?gm5043) (end of section 9.2) which references a paper in one of the old LURE reports (difficult to find, but Tom has the PDF).
Slow prediction motivated the addition of `--cell-parameters-only` in 46baa4da52b939db5e8a01271befb9f5e704b3b7. For online cell monitoring, no prediction is obviously the fastest possible solution. Nevertheless, there's a big speed improvement to be made here.CrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/69Ambigator doesn't work with non-integer ambiguity operators2022-11-18T17:18:20+01:00Thomas WhiteAmbigator doesn't work with non-integer ambiguity operatorsAmbigator uses SymOpList, and hence IntegerMatrix, internally. That's all that was available when it was written! However, now we have RationalMatrix. And, it turns out, there are indexing ambiguities which require non-integer matrice...Ambigator uses SymOpList, and hence IntegerMatrix, internally. That's all that was available when it was written! However, now we have RationalMatrix. And, it turns out, there are indexing ambiguities which require non-integer matrices. Example: orthorhombic C-centered with a and b axis ratio just right to make it look hexagonal.
Ambigator could be relatively easily converted to use a single RationalMatrix to represent the ambiguity operator.https://gitlab.desy.de/thomas.white/crystfel/-/issues/62Reindexing option for Unit Cell Explorer2022-02-21T16:09:15+01:00Thomas WhiteReindexing option for Unit Cell ExplorerA feature suggestion: The cell explorer could simplify the graphs by offering to reindex A, B and C cells all to C (or B, or I suppose A if the user wants). The same could be done for H and R.A feature suggestion: The cell explorer could simplify the graphs by offering to reindex A, B and C cells all to C (or B, or I suppose A if the user wants). The same could be done for H and R.https://gitlab.desy.de/thomas.white/crystfel/-/issues/57HDF5 peak lists with online data2024-01-02T15:31:07+01:00Thomas WhiteHDF5 peak lists with online dataAs noted in the [documentation for online data](https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/online.rst), `--peaks=hdf5` cannot yet be combined with online data from ZMQ or ASAP::O. This is because `image_read...As noted in the [documentation for online data](https://gitlab.desy.de/thomas.white/crystfel/-/blob/master/doc/articles/online.rst), `--peaks=hdf5` cannot yet be combined with online data from ZMQ or ASAP::O. This is because `image_read_peaks` (and descendants) take a filename. This needs to be generalised to handle a data block.CrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/40Prepare for GTK4 migration2023-03-02T12:44:21+01:00Thomas WhitePrepare for GTK4 migrationThis is not urgent, but at some point we will need to upgrade to GTK4. There are some preparatory steps which could be done earlier, to make the switch easy when the time comes. They are explained here:
https://developer.gnome.org/gtk4...This is not urgent, but at some point we will need to upgrade to GTK4. There are some preparatory steps which could be done earlier, to make the switch easy when the time comes. They are explained here:
https://developer.gnome.org/gtk4/4.0/gtk-migrating-3-to-4.htmlhttps://gitlab.desy.de/thomas.white/crystfel/-/issues/39Work out correct lattice type and unique axis after cell transformation2021-05-17T11:51:33+02:00Thomas WhiteWork out correct lattice type and unique axis after cell transformationCurrently, `cell_transform_rational` just marks the output cell as triclinic with unknown unique axis. It should be possible to algebraically work out the real lattice type and unique axis. No cheating by comparing axis lengths and ang...Currently, `cell_transform_rational` just marks the output cell as triclinic with unknown unique axis. It should be possible to algebraically work out the real lattice type and unique axis. No cheating by comparing axis lengths and angles, because that's not very general. A general solution is probably worth a paper.https://gitlab.desy.de/thomas.white/crystfel/-/issues/35Automatically determine symmetry for merging and ambiguity resolution2024-01-02T15:32:39+01:00Thomas WhiteAutomatically determine symmetry for merging and ambiguity resolutionCrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/30Automatically determine unit cell for FoMs and export2024-01-02T15:32:25+01:00Thomas WhiteAutomatically determine unit cell for FoMs and exportCurrently, the user has to create a "final" unit cell file and select it when calculating FoMs or exporting data. The GUI has all the information required to work out the cell for itself.Currently, the user has to create a "final" unit cell file and select it when calculating FoMs or exporting data. The GUI has all the information required to work out the cell for itself.CrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/23Progress bar / abort option for "index one pattern"2024-01-02T15:33:25+01:00Thomas WhiteProgress bar / abort option for "index one pattern"Indexing one pattern is usually very quick, especially thanks to recent speed improvements. However, sometimes it still takes a long time. A progress bar and cancel option would be a big help.
This is quite difficult, though. The pro...Indexing one pattern is usually very quick, especially thanks to recent speed improvements. However, sometimes it still takes a long time. A progress bar and cancel option would be a big help.
This is quite difficult, though. The processing would have to be done in a different thread, or even a different process.CrystFEL 0.12.0https://gitlab.desy.de/thomas.white/crystfel/-/issues/22Automatic methods section writer2023-09-29T09:02:27+02:00Thomas WhiteAutomatic methods section writerPeople seem to struggle with getting all the relevant information into methods sections. We can help here, by adding a way to automatically write about a paragraph of text based on a template. The new "harvest" files can help here.People seem to struggle with getting all the relevant information into methods sections. We can help here, by adding a way to automatically write about a paragraph of text based on a template. The new "harvest" files can help here.https://gitlab.desy.de/thomas.white/crystfel/-/issues/21Reconstruct SLURM jobs after GUI restart2021-04-27T16:14:30+02:00Thomas WhiteReconstruct SLURM jobs after GUI restartIt should be possible to start a load of SLURM jobs, close the GUI, and come back later. Currently, the running jobs will be completely lost. It should be possible to reconstruct the jobs on re-start.It should be possible to start a load of SLURM jobs, close the GUI, and come back later. Currently, the running jobs will be completely lost. It should be possible to reconstruct the jobs on re-start.https://gitlab.desy.de/thomas.white/crystfel/-/issues/10Retrieve values from CBF headers2021-04-27T14:43:00+02:00Thomas WhiteRetrieve values from CBF headersCurrently, metadata retrieval, e.g. for the X-ray wavelength, only works from HDF5 fields and stream input. CBF should be added to this list. This is more difficult than it seems, because CBF headers are a bit of a messy mixture of pro...Currently, metadata retrieval, e.g. for the X-ray wavelength, only works from HDF5 fields and stream input. CBF should be added to this list. This is more difficult than it seems, because CBF headers are a bit of a messy mixture of properly formatted CIF headers and blocks of custom Dectris headers (with various inconsistent labels). There'd need to be some way of defining exactly where a value should come from.
Although it's a big omission, surprisingly no-one seems to be asking for this feature at the moment. Therefore it's kind of lower priority. Disagree? Comment here.https://gitlab.desy.de/thomas.white/crystfel/-/issues/4stream_grep needs to be updated for multi-crystal indexing2024-02-27T10:10:18+01:00Thomas Whitestream_grep needs to be updated for multi-crystal indexingMigrated from Jira (CRYS-228).Migrated from Jira (CRYS-228).