Initial release

.gitattributes

@@ -0,0 +1,40 @@
+*.7z filter=lfs diff=lfs merge=lfs -text
+*.arrow filter=lfs diff=lfs merge=lfs -text
+*.bin filter=lfs diff=lfs merge=lfs -text
+*.bz2 filter=lfs diff=lfs merge=lfs -text
+*.ckpt filter=lfs diff=lfs merge=lfs -text
+*.ftz filter=lfs diff=lfs merge=lfs -text
+*.gz filter=lfs diff=lfs merge=lfs -text
+*.h5 filter=lfs diff=lfs merge=lfs -text
+*.joblib filter=lfs diff=lfs merge=lfs -text
+*.lfs.* filter=lfs diff=lfs merge=lfs -text
+*.mlmodel filter=lfs diff=lfs merge=lfs -text
+*.model filter=lfs diff=lfs merge=lfs -text
+*.msgpack filter=lfs diff=lfs merge=lfs -text
+*.npy filter=lfs diff=lfs merge=lfs -text
+*.npz filter=lfs diff=lfs merge=lfs -text
+*.onnx filter=lfs diff=lfs merge=lfs -text
+*.ot filter=lfs diff=lfs merge=lfs -text
+*.parquet filter=lfs diff=lfs merge=lfs -text
+*.pb filter=lfs diff=lfs merge=lfs -text
+*.pickle filter=lfs diff=lfs merge=lfs -text
+*.pkl filter=lfs diff=lfs merge=lfs -text
+*.pt filter=lfs diff=lfs merge=lfs -text
+*.pth filter=lfs diff=lfs merge=lfs -text
+*.rar filter=lfs diff=lfs merge=lfs -text
+*.safetensors filter=lfs diff=lfs merge=lfs -text
+saved_model/**/* filter=lfs diff=lfs merge=lfs -text
+*.tar.* filter=lfs diff=lfs merge=lfs -text
+*.tar filter=lfs diff=lfs merge=lfs -text
+*.tflite filter=lfs diff=lfs merge=lfs -text
+*.tgz filter=lfs diff=lfs merge=lfs -text
+*.wasm filter=lfs diff=lfs merge=lfs -text
+*.xz filter=lfs diff=lfs merge=lfs -text
+*.zip filter=lfs diff=lfs merge=lfs -text
+*.zst filter=lfs diff=lfs merge=lfs -text
+*tfevents* filter=lfs diff=lfs merge=lfs -text
+nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/progress.png filter=lfs diff=lfs merge=lfs -text
+nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/progress.png filter=lfs diff=lfs merge=lfs -text
+nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/progress.png filter=lfs diff=lfs merge=lfs -text
+nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/progress.png filter=lfs diff=lfs merge=lfs -text
+nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/progress.png filter=lfs diff=lfs merge=lfs -text

.gitignore

@@ -0,0 +1,3 @@
+# The parent research repository ignores generated images globally. These
+# original nnU-Net plots are intentional public provenance artifacts.
+!**/logs/progress.png

LICENSE

@@ -0,0 +1,41 @@
+BodyCompositionCT-ResEncL license notice
+
+Licensed Material
+-----------------
+
+To the extent that copyright or similar rights subsist, the original
+BodyCompositionCT-ResEncL trained model weights, the included training and
+evaluation records, and original documentation and metadata distributed in
+this repository are available under the Creative Commons Attribution 4.0
+International Public License (CC BY 4.0).
+
+Canonical license URL:
+https://creativecommons.org/licenses/by/4.0/
+
+Full legal code:
+https://creativecommons.org/licenses/by/4.0/legalcode.en
+
+Attribution
+-----------
+
+When sharing the Licensed Material, including a modified version, provide
+appropriate credit, link to CC BY 4.0, and indicate whether changes were made.
+A suitable attribution is:
+
+  BodyCompositionCT-ResEncL by Felix O. Hofmann,
+  https://huggingface.co/fhofmann/BodyCompositionCT-ResEncL,
+  licensed under CC BY 4.0; changes indicated where applicable.
+
+Scope exclusions
+----------------
+
+This notice does not apply to, and does not relicense, source CT images,
+third-party source annotations, upstream model weights, or third-party
+software. Those materials are not distributed in this repository. See
+THIRD_PARTY_NOTICES.md for provenance and upstream terms.
+
+No executable training or inference code is covered by this notice.
+
+The full CC BY 4.0 legal code controls if this notice and the license differ.
+The Licensed Material is provided without warranties, to the extent permitted
+by law.

MODEL_ARTIFACTS.sha256

@@ -0,0 +1,8 @@
+8de2ff6b2b4229312517acf9bfd3dd4d23cc7ebedc48c0ad8e36bd3ba10e7f1d  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/dataset.json
+1790c1825529e29de536c73f66f5fd8ee12b9812271e93434e76a000c1e7cae0  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/plans.json
+03db52a2a175b6c46f0f8ba75f3d969a3d33e8f567ced072cfa4bc1e8715a82b  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/postprocessing.pkl
+d4eb90614fef10c2eb9aad574d4cb3517511d224f2f2da897fe4e4618513cd77  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/checkpoint_final.pth
+f520d3293d667986dd89fae115e3602af9f40c98413a08808d1cbb0ce08e074c  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/checkpoint_final.pth
+eeca2b9d95426147c6d533ee82ea43c215e33a785faaeb2443e5def4328bf42c  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/checkpoint_final.pth
+8a2d7fc6a32dccad3a43cd7fcad392b38df490303baf3842e8ee57587bcfbf4a  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/checkpoint_final.pth
+9d4c69c505961643bc97490e7a162258ac0bd3d2d17a6f58b8900d2143946231  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/checkpoint_final.pth

README.md

@@ -0,0 +1,208 @@
+---
+pipeline_tag: image-segmentation
+license: cc-by-4.0
+tags:
+- medical
+- ct
+- body-composition
+- 3d-segmentation
+- nnunet
+- residual-encoder
+---
+
+# BodyCompositionCT-ResEncL
+
+BodyCompositionCT-ResEncL is a 3D nnU-Net v2 model for segmenting native
+body-composition compartments and supporting anatomical structures in CT. This
+is a five-fold ResEnc L ensemble. For single-checkpoint inference, see
+[BodyCompositionCT-ResEncM](https://huggingface.co/fhofmann/BodyCompositionCT-ResEncM).
+
+## Model overview
+
+| Property | Value |
+| --- | --- |
+| Task | Mutually exclusive 3D semantic segmentation |
+| Input | One CT volume |
+| Architecture | nnU-Net v2, `3d_fullres`, ResEnc L |
+| Checkpoints | Five cross-validation folds, used as an ensemble |
+| Training set | 1,656 CT scans |
+| Native output | Background plus seven foreground labels |
+| License | CC BY 4.0 |
+
+## Native output labels
+
+| ID | Label | Description |
+| ---: | --- | --- |
+| 0 | Background | |
+| 1 | Muscle | Muscle compartment |
+| 2 | Bone | Bone |
+| 3 | Subcutaneous | Subcutaneous compartment |
+| 4 | Abdominal | Abdominal compartment, with organs and larger vessels excluded |
+| 5 | Thoracic | Thoracic compartment, with organs and larger vessels excluded |
+| 6 | Heart | Heart |
+| 7 | Lungs | Lungs |
+
+Labels 1, 3, 4, and 5 denote anatomical compartments rather than final
+attenuation-defined tissue masks. The muscle compartment can contain inter- and
+intramuscular connective tissue, smaller vessels, and larger fat-attenuation
+regions. IMAT is not a separate model class; CT-visible IMAT can be derived
+downstream within label 1 using HU-thresholding. All downstream tissue
+measurements should state their HU range, cleanup rules, and source compartment.
+
+The machine-readable mapping is stored in
+[dataset.json](nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/dataset.json).
+
+## Training data and target construction
+
+The training set contains 750 CT scans from
+[SAROS](https://doi.org/10.1038/s41597-024-03337-6) and 906 from the
+[TotalSegmentator v2 dataset](https://doi.org/10.5281/zenodo.10047292).
+Another 332 cases (150 SAROS and 182 TotalSegmentator) were reserved and were
+not used for training.
+
+Targets were constructed from dense
+[BOA](https://doi.org/10.1097/RLI.0000000000001040) Task542 body-region
+predictions, sparse expert-reviewed SAROS labels, and TotalSegmentator-derived
+anatomical masks. Reviewed SAROS labels replaced predictions on annotated
+slices. TotalSegmentator muscle, bone, lung, tracheal, body-trunk, organ,
+vessel, and neural masks supplied corrections and exclusions. HU-guided
+candidate selection, morphological cleanup, model-assisted completion, and
+manual review produced the final seven foreground labels.
+
+Only the open TotalSegmentator `total` and `body` tasks were used for target
+construction; the separately licensed `tissue_types` and `vertebrae_body`
+tasks were not used.
+
+Training logs, plans, configuration snapshots, progress plots, and internal
+cross-validation summaries are included with the model.
+
+## Inference
+
+The model uses the standard nnU-Net v2 results layout. The commands below were
+tested with nnU-Net v2 2.5.2. Input files must be unwindowed CT NIfTI volumes
+containing Hounsfield-unit values and named `<case>_0000.nii.gz`. Do not
+normalize or resample them manually; nnU-Net applies the preprocessing stored
+in the plans.
+
+For integration with preprocessing and downstream tissue derivation, see the
+[BodyComposition pipeline](https://github.com/fohofmann/BodyComposition).
+
+Set the nnU-Net results directory and download the model:
+
+```bash
+export nnUNet_results="/path/to/nnUNet_results"
+
+hf download fhofmann/BodyCompositionCT-ResEncL \
+  --local-dir "${nnUNet_results}/Dataset611_Tissue"
+```
+
+Optionally verify the model artifacts:
+
+```bash
+cd "${nnUNet_results}/Dataset611_Tissue"
+sha256sum -c MODEL_ARTIFACTS.sha256
+```
+
+Run five-fold ensemble inference:
+
+```bash
+nnUNetv2_predict \
+  -d Dataset611_Tissue \
+  -i INPUT_FOLDER \
+  -o OUTPUT_FOLDER \
+  -f 0 1 2 3 4 \
+  -tr nnUNetTrainer \
+  -c 3d_fullres \
+  -p nnUNetResEncUNetLPlans
+```
+
+Cross-validation selected largest-component postprocessing for label 6
+(heart):
+
+```bash
+MODEL_FOLDER="${nnUNet_results}/Dataset611_Tissue/nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres"
+
+nnUNetv2_apply_postprocessing \
+  -i OUTPUT_FOLDER \
+  -o OUTPUT_FOLDER_POSTPROCESSED \
+  -pp_pkl_file "${MODEL_FOLDER}/crossval_results_folds_0_1_2_3_4/postprocessing.pkl" \
+  -plans_json "${MODEL_FOLDER}/plans.json" \
+  -dataset_json "${MODEL_FOLDER}/dataset.json"
+```
+
+With NumPy 2.4 or newer, use `acvl-utils>=0.2.6,<0.3` for this
+postprocessing step.
+
+The checkpoints and postprocessing file are serialized PyTorch/Python
+artifacts. Load them only from a trusted copy of this repository.
+
+## Evaluation status
+
+The included
+[cross-validation summary](nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/summary.json)
+reports internal five-fold cross-validation across the 1,656 training cases;
+each case was evaluated by a fold that did not train on it.
+
+## Intended use and limitations
+
+The model is intended to provide compartment masks for research pipelines,
+including downstream body-composition measurements derived with CT attenuation thresholds.
+
+- It is not intended for diagnosis, treatment decisions, or stand-alone
+  clinical measurement.
+- Training targets combine automated segmentations, deterministic corrections,
+  and manual review; residual source-model and labeling errors can remain.
+- Performance may vary across populations, scanners, acquisition protocols,
+  contrast phases, artifacts, implants, and uncommon anatomy.
+- The targets focus on the body trunk; extremity use is out of scope.
+- External validation is required for each intended population and
+  measurement definition.
+
+## License and attribution
+
+The trained weights, model documentation, metadata, and included
+training/evaluation records are released under
+[CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Reuse requires
+appropriate credit, a link to the license, and an indication of changes. See
+[LICENSE](LICENSE) for the attribution format.
+
+Training source data, source annotations, upstream checkpoints, and
+third-party software are not redistributed and retain their original terms:
+
+| Upstream material | Role | Terms |
+| --- | --- | --- |
+| SAROS labels and split metadata | Sparse reviewed labels | CC BY 4.0 |
+| SAROS source CT scans | Training images | Collection-specific TCIA terms, including restricted, CC BY, and CC BY-NC collections |
+| BOA Task542 weights | Dense initial body-region predictions | MIT |
+| TotalSegmentator v2 dataset | Training CT scans and anatomical labels | CC BY 4.0 |
+| TotalSegmentator `total` and `body` tasks | Corrections and exclusions | Apache-2.0 |
+| nnU-Net v2 | Training framework and architecture | Apache-2.0 |
+
+See [THIRD_PARTY_NOTICES.md](THIRD_PARTY_NOTICES.md) for source links and the
+complete provenance and license summary.
+
+## References
+
+- Hofmann FO et al. *Validation of body composition parameters extracted via
+  deep learning-based segmentation from routine computed tomographies.*
+  Scientific Reports (2025).
+  https://doi.org/10.1038/s41598-025-96238-6
+- Haubold J et al. *SAROS: A dataset for whole-body region and organ
+  segmentation in CT imaging.* Scientific Data (2024).
+  https://doi.org/10.1038/s41597-024-03337-6
+- Haubold J et al. *BOA: A CT-Based Body and Organ Analysis for Radiologists at
+  the Point of Care.* Investigative Radiology (2023).
+  https://doi.org/10.1097/RLI.0000000000001040
+- Wasserthal J et al. *TotalSegmentator: Robust Segmentation of 104 Anatomic
+  Structures in CT Images.* Radiology: Artificial Intelligence (2023).
+  https://doi.org/10.1148/ryai.230024
+- Isensee F et al. *nnU-Net: a self-configuring method for deep
+  learning-based biomedical image segmentation.* Nature Methods (2021).
+  https://doi.org/10.1038/s41592-020-01008-z
+- Isensee F et al. *nnU-Net Revisited: A Call for Rigorous Validation in 3D
+  Medical Image Segmentation.* arXiv (2024).
+  https://doi.org/10.48550/arXiv.2404.09556
+
+Questions and feedback are welcome in the repository's
+[Community](https://huggingface.co/fhofmann/BodyCompositionCT-ResEncL/discussions)
+section.

THIRD_PARTY_NOTICES.md

@@ -0,0 +1,24 @@
+# Third-party provenance and licenses
+
+This repository distributes BodyCompositionCT-ResEncL trained weights,
+documentation, metadata, and nnU-Net training and evaluation records. Source
+CT images, source annotations, upstream checkpoints, and third-party code are
+not redistributed.
+
+| Material or software | Use | Upstream terms |
+| --- | --- | --- |
+| SAROS segmentations and split metadata | Sparse reviewed body-region labels used during target construction | CC BY 4.0; [dataset](https://doi.org/10.25737/SZ96-ZG60), [paper](https://doi.org/10.1038/s41597-024-03337-6) |
+| SAROS source CT images | Training images | Collection-specific TCIA terms, including TCIA Restricted, CC BY 3.0, CC BY-NC 3.0, and CC BY 4.0; not redistributed. [SAROS/TCIA page](https://wiki.cancerimagingarchive.net/pages/viewpage.action?pageId=157287899) |
+| BOA Task542 body-region weights | Dense body-region predictions used during target construction | MIT; [weights](https://doi.org/10.5281/zenodo.7918824), [paper](https://doi.org/10.1097/RLI.0000000000001040) |
+| TotalSegmentator v2 CT dataset | Training CT images and anatomical labels | CC BY 4.0; [dataset](https://doi.org/10.5281/zenodo.10047292) |
+| TotalSegmentator `total` and `body` tasks | Anatomical corrections and exclusions | Apache-2.0; [repository](https://github.com/wasserth/TotalSegmentator), [paper](https://doi.org/10.1148/ryai.230024) |
+| nnU-Net v2 and residual-encoder configuration | Training framework and model architecture | Apache-2.0; [repository](https://github.com/MIC-DKFZ/nnUNet), [nnU-Net paper](https://doi.org/10.1038/s41592-020-01008-z), [residual-encoder paper](https://doi.org/10.48550/arXiv.2404.09556) |
+
+CC BY 4.0 applies to the BodyCompositionCT trained weights, documentation,
+metadata, and included training/evaluation records. Source data, source labels,
+upstream checkpoints, and third-party code are not included and remain under
+their original terms. The listed upstream terms do not impose a
+non-commercial or share-alike condition on this model package.
+
+The target construction used the open TotalSegmentator `total` and `body`
+tasks, not the separately licensed `tissue_types` or `vertebrae_body` tasks.

TRAINING_ARTIFACTS.sha256

@@ -0,0 +1,97 @@
+a50a652f4400e8aee0bb1a3b716fd55c5ff42aa51af2f4eb404058b42cbf9b56  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/postprocessed/summary.json
+dce71996932bb34e4cbbafd44e03ace1f82ab4258062c42f1377b1cc6166150e  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/postprocessing.json
+7852a00bb34b56a66877eb413b32c38a7e168e04e89afc5494e9749085377b67  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/summary.json
+f5f73926aa5509fd0dbfcb38b3d9b553c6e6aabd27cd27da5e772f9c3c8ac043  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/debug.json
+f0f7d2a390a984f01327a8cf6854bb19502fd392ad02e49d25ba58cc6a2bc6b1  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/progress.png
+e842ca57175ae0c8186bd149349ad8711a79cf0b3adebafb16f1c6fc3c28794d  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_10_06_01_54.txt
+2d8c126874ac3c443cbf18744cdc2b1c730c3ff80bbece85107678b125d0e9cb  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_10_14_01_21.txt
+8668e7ed31c18af89c6af446fb7d96f46a4b5fb8c5b8d8e3a85ddeb031786221  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_10_22_43_26.txt
+899037d9de50797a959e98ae1cac0e9dda3abd2d8a19b2850236b85a3122b4a9  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_11_06_42_34.txt
+642d3cbc29c2fa816248f3946febd70e461c6715a62787cb067f84a8372d4227  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_11_14_42_12.txt
+9ae37da209ef348cf0d1beef00d90d6e065cbab8ef74a168a980aef97b28fcbf  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_4_16_32_30.txt
+bec3f8c2964c93340ea76b98d1c13e6fe7460a566eb340526550358d0a8076f5  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_5_00_32_01.txt
+b6085e8730d7591f3bfc183513f3207e92c36ac7056f29d1d1bb6108df901aa7  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_5_08_31_30.txt
+474d3700dfbe8b6e7b9384e669908b04991f88b930893975686dbbcbf1c8ff42  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_5_16_30_45.txt
+1bd71ad1a035ef93c9dd13e71c2c4212ed5e0f2f6456d013dc811fbc157e264e  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_6_00_30_15.txt
+c86adfc3e4f145580eab7b06ce194e3c88929755c1c1209903ee5073fef3c6ad  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_6_08_29_45.txt
+b4674bb1d65debb48689e804363e2d444613e422a9efde9a32ff8fa4a5255769  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_7_14_05_16.txt
+ec19be9c891048fbbac561f0b4676eafd0606191dbeb60b2ef7ef6364fd4a49e  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_7_22_04_47.txt
+0467161b67044210715189158a5cfdbd85bc2f6c2a381d2f46e454dce3854413  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_8_06_04_18.txt
+877c971a9f2d73778d7fd514d142180b94c1479cdc429704f3b3515412fb468f  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_8_14_03_50.txt
+b8fdf4f012e021694dce2dda716054d53e46326cb738bf9c5348243bc184faa7  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_8_22_03_23.txt
+4788f8dadf37788ce455c0fe9b0edd2d74f30af8b2239c4560ecc65b2bf03d8a  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_9_06_02_53.txt
+ea1792ef878334505f57e189cb08fd6bbb0607a49a44ae99713c07d3b9a2fb65  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_9_14_03_19.txt
+7c72769ea23702b023b060a4361d6f8b724531d48e00236bedbf744aecfd3085  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_9_22_02_26.txt
+5e888a7525bd77288dede3e4001f81b55ed0171fd5b435a9313dc199a548e857  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/debug.json
+8da68332e056494c7d97d305d52eda20b11f0475aec7eeed46d992799cacbcee  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/progress.png
+d4c6e2ffe28781e24a6385bb643f21b0d5849ff1b7a73221ccc074fd11197cf8  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_10_00_29_55.txt
+b9cca418a90ca90e70fb169069d82f9cb5731e3427bcbb59c90dbfb33d0e5b08  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_10_08_29_27.txt
+09b21df38d03c178f811df47cfbde158d8a5cde1581aaf4205356cab06ad5e7b  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_4_16_32_31.txt
+1aa863b1eed19537e8a49d82189c110ad6a3e85ef14f736b8e0e861db5dff592  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_5_00_32_01.txt
+dd44c61182dcfb0b8509f05375309003bf97559747b51a513daeb6b81d6a89c4  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_5_08_31_30.txt
+b98aac8e9fc0e4c534d52a5d3ca4c1c91901e41593bcea6e4dbcaed3a32ac504  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_5_16_30_44.txt
+f4d03b91f60c1e091ff45dc1f4bf8c093bdb0c6fecbc9a99069c290a452daee5  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_6_00_30_15.txt
+2ddd90ba1dae95735dc1c1baa8f9e082a723f9c3590ca2fd9108d201004c9929  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_6_08_29_45.txt
+188f4b8a699da014f654e2d0e37f11caa9dc35b6eb2af248680dbf1444d60688  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_6_17_12_15.txt
+dd56dbc0e6beb9c9636a5068aa21b2d9302cbe4abc486df81dc0bead7efd1c84  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_7_14_05_16.txt
+51e74b8acd65346c34976e753c9c15085175750e79864aaccb9e09e8bac4c652  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_7_22_04_47.txt
+51da9e0010d7a837eab7e555abdeed9ef6cdb2904c8ef3da89bbb027b63e87f9  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_8_06_04_19.txt
+b2905c6541205ee40fc6dd98e84c8350c9973524f2cea1e49cfda7efb1e12739  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_8_14_03_51.txt
+b24971b77208fa82a8fd5696b920576131ea281d6014596d59d6b299ae7d55ba  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_8_22_03_23.txt
+347f4b1af773b4a30bba1fd2a1b2fc941df81fbd82196fc34cca484cd649161c  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_9_06_02_53.txt
+3d2d9e61c94058d1b83b05d908a46791ace1c7e8cd0c0de3a90eb2a10df640ba  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_9_16_30_49.txt
+51e289ec93bb50b66a46a9a1346059663c813f2203e75dfd3d7cd5de8f3c813c  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/debug.json
+369223df1451bfe1ffce6a0c79dbab18e5cad51a9c080ca87203a663f52e2568  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/progress.png
+d047d3472603bfbdf5b0a39da480b6312be302bafee0ae90736160f7e260d04f  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_10_00_29_57.txt
+f7b6f31024d2235029cd24e47ed9128c9ad9be38a4889a080bf9f920d632c3bf  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_10_08_29_27.txt
+030e3301c6239c9c5e622416c2a7ef7494bbc2481699d8e4454fb122632ed203  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_10_16_28_32.txt
+8198a001c108b673337e831e829f2b942b92dc358f0ecb292d7b14bd18e0dc93  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_11_00_05_58.txt
+60cf20ab89d27cf88d31614321d537ad006c6dadfcb61a1374931dcf64eebf46  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_4_16_32_33.txt
+a0787ba95c253a582e157409f1490d1b352cff0992db3a55b75eee0eb3821cdb  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_5_00_32_01.txt
+dd91697b6d169764d2b1521a8491f45a79278a908c1301c0da9a9f5667655c57  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_5_08_31_30.txt
+3b8e3b429f70d9d5bc2b2faa330f3271e653f617d148f5dd2998db8991a2347a  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_5_16_30_44.txt
+7d95199885857ba7c06c3b755ef5e5411ca3e9a796326fbbdf6486d6eae1f402  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_6_00_30_15.txt
+a8a06ecf06e566432aabbdfc41d1bb6a2a1bea76e8752ef2feb93d3062a45456  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_6_08_29_45.txt
+d4e6e03b387dbad0f538840e748d0899cb800010ef2f7f93b640fba1c96bfcca  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_7_14_05_16.txt
+8e1aaeaf45c5467bd45b17310ce3ae09971caa1e487f99e3510b82d1171f021f  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_7_22_04_47.txt
+c70c99bcf562a859cf655bd1c1af03fbf61ca598b554c9af81c6dff318dac3d8  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_8_06_04_18.txt
+cd13f24fa0a6202b92eb7cfb66827162fe16297b3f765711614d8fc2a3ad23af  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_8_14_03_50.txt
+36a49f5e63ab5d0851bfb4e2214f8a7a824966c19215a279af750b053334f4de  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_8_22_03_23.txt
+e2d501956044b2427756afe259f458afce0a6d4ed379070633fc7f67781b0608  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_9_06_02_54.txt
+66e03b2625fd54ddc41df0902400563609adb3e6dd6197d269e099371373654e  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_2/logs/training_log_2024_9_9_16_30_50.txt
+2fdcd4d5b4e09eda10123776189be3653f6b5c38fc47be3ba8fa583c461f6a01  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/debug.json
+0d4b4482fdeb6679f999f8adb43ebce2973eaabb53f654d4493c97cd5b9f3708  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/progress.png
+22b1b183898ad463bd2c5434a60b2ec0ba40a22ecf92d8911347085e2990f351  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_10_00_53_20.txt
+92c19051aad2f1a3935e861090fd8ca879782a224916bdc6158a670817ff7a42  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_4_16_32_36.txt
+e17a065602d5b78c7e68bff3908cc64a46bf91aeb274a95ef6bc4b0e36256ce8  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_5_00_32_01.txt
+efdef825b6010e8cdcbdce34fed2f44f00212ef6f80e21a4843dc85e3ced8951  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_5_08_31_29.txt
+120a5ff5451ddfd410a946a8814df28a4a41049574a6454bffd5879e9674253b  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_5_16_30_45.txt
+4a499bd083755db84da634f9253577dca2f63005cd854af6e11377ce5b55b480  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_6_00_30_15.txt
+5d7af324f4fb3304910db8910546ba90d3d04d494dee4ef3ffa1edf2a3982677  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_6_08_29_45.txt
+d8d83f3ae8dc58f3dc7032909f4adb6a630b813cbaafee02634dd2a4a2f83530  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_6_17_12_21.txt
+c8d1cd61a3195131046721d31b68717087a8707cf307c74585ddeb4c5d0917b2  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_7_14_05_16.txt
+01f7bdbe7ce9e7c446a2c86f2725acdd65db2b51380ddcc3303c723f2310e6c4  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_7_22_04_47.txt
+287822a7c40dfd138f04d96fea7206a678d814d50fbb54f066b0530d67722574  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_8_06_04_18.txt
+6706ab602d8962466b5b5a438ffd8625f2efcc711961ff9187e36af037b9d238  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_8_14_03_51.txt
+2307b4dc965d7bd94e16c64d550fed5079284e8ceebf9507b89418951f0ac03f  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_8_22_03_23.txt
+c1e8d03bc8acaff1c0c5ceb07b8f21a5f91641548e238807514eb39391d3941c  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_9_06_02_53.txt
+d57fa0d3515d9ebc8dd9b80c5a307184ef339475104b5de7b8386c14f95d448d  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_3/logs/training_log_2024_9_9_16_54_15.txt
+f7a80adbd226b0bbada8d94187315d5708b8cb1f5daa9a2c1c2f057ad6f5480f  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/debug.json
+52274fd851730067db78601548e4ae1490545a8e2a493288137cc255451f3721  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/progress.png
+88f7c50c754d3cbe97235f065de379b2596e16d0b0dff48012beb82d13c90b84  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_10_00_53_30.txt
+6125ba592761997ad491fa1430e04eed7314ec5c06dc7517480117d4ee9277d7  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_10_08_52_56.txt
+c1188b3b72ebe3fcc18a2a418681af183d8bda8857ca10705ceed8aee1838d57  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_10_16_52_27.txt
+ee994d317b6b3dc49bf52e3ff8a1175f3b611df33853726fbd6089e6186e7cd7  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_10_23_53_16.txt
+a59a62a014039e08302a62addf69958726f070f6aad257057ad3d1682301bb74  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_4_16_32_40.txt
+47a4b8f283e345f8898110f69d5e4b237bec2adbd9a7a08f6f770b541eb1192a  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_5_00_32_01.txt
+010c4e42d58f0396c282b3c9561d33e6a5e8e55d9f9696456537fdf1ff51866b  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_5_08_31_30.txt
+2168c2c1f4a0612d4be04437a628a25624bb935b029c0226fdbb7f7d1d6b4195  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_5_16_30_44.txt
+cd78be217359ba5706fa0d57790cdf44a81ba059c90ee4de50b56829b54d581e  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_6_00_30_16.txt
+a67de991f40698773105698d3723017513149b3e512bb9fdaaee678674959d4d  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_6_08_29_45.txt
+18fe0230faad519953ecf84267b67bd781006c0f9ad9d2d1373a1b482771ec1b  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_6_17_12_24.txt
+a9fdd962a6f7afa3ad253c97f7f8341a7e0bee63a64a9c1f9b1ec7c025812e19  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_7_14_05_16.txt
+97acc275669a5c5e908821f6735e16b939646bc6e9d445b43a13119bb38a0b3a  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_7_22_04_48.txt
+aac84c1f03d242dae69e0b6fa1d4959834b13c2c479df1751efb29c2a264dec4  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_8_06_04_18.txt
+52e2e40868f5949b4d4a0e312aebc2cb6d45035a1c0308b2593faf15e95404c1  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_8_14_03_50.txt
+68471eb1b45384891e286524e6be613fa70e06635b02ddcd7234590a83fab9cc  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_9_06_03_47.txt
+8bab25b3e8fe6ff7b0766510bb211139d945cf05dd9ec0b1466fa44973d698fd  nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_4/logs/training_log_2024_9_9_16_54_15.txt

config.json

@@ -0,0 +1,19 @@
+{
+  "schema_version": 1,
+  "model_format": "nnUNet v2",
+  "task": "3d_semantic_segmentation",
+  "dataset_id": "Dataset611_Tissue",
+  "trainer": "nnUNetTrainer",
+  "configuration": "3d_fullres",
+  "plans_identifier": "nnUNetResEncUNetLPlans",
+  "folds": [
+    0,
+    1,
+    2,
+    3,
+    4
+  ],
+  "dataset_json": "nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/dataset.json",
+  "plans_json": "nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/plans.json",
+  "checkpoint_filename": "checkpoint_final.pth"
+}

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/postprocessing.json

@@ -0,0 +1,178 @@
+{
+    "input_folder": {
+        "foreground_mean": {
+            "Dice": 0.9478300201361252,
+            "FN": 24221.183747412004,
+            "FP": 23231.3088336784,
+            "IoU": 0.9111582927428803,
+            "TN": 20772368.677191164,
+            "TP": 875227.8290200139,
+            "n_pred": 898459.1378536922,
+            "n_ref": 899449.0127674259
+        },
+        "mean": {
+            "1": {
+                "Dice": 0.9663381582297933,
+                "FN": 51781.09178743961,
+                "FP": 54967.756642512075,
+                "IoU": 0.9378369176697793,
+                "TN": 19720238.92753623,
+                "TP": 1868061.222826087,
+                "n_pred": 1923028.979468599,
+                "n_ref": 1919842.3146135265
+            },
+            "2": {
+                "Dice": 0.957884407011915,
+                "FN": 21668.210144927536,
+                "FP": 22863.38345410628,
+                "IoU": 0.9217993012824045,
+                "TN": 21059716.478864733,
+                "TP": 590800.9263285024,
+                "n_pred": 613664.3097826086,
+                "n_ref": 612469.13647343
+            },
+            "3": {
+                "Dice": 0.9672303238506204,
+                "FN": 45615.25,
+                "FP": 36531.213768115944,
+                "IoU": 0.940791967283391,
+                "TN": 19712272.719806764,
+                "TP": 1900629.8152173914,
+                "n_pred": 1937161.0289855073,
+                "n_ref": 1946245.0652173914
+            },
+            "4": {
+                "Dice": 0.9271283978454077,
+                "FN": 31290.937198067633,
+                "FP": 29929.147946859903,
+                "IoU": 0.8753949303608027,
+                "TN": 20927857.83574879,
+                "TP": 705971.0778985508,
+                "n_pred": 735900.2258454106,
+                "n_ref": 737262.0150966183
+            },
+            "5": {
+                "Dice": 0.8775996178802443,
+                "FN": 9464.621376811594,
+                "FP": 7677.018719806763,
+                "IoU": 0.7946187000311267,
+                "TN": 21599642.222222224,
+                "TP": 78265.13647342996,
+                "n_pred": 85942.15519323671,
+                "n_ref": 87729.75785024154
+            },
+            "6": {
+                "Dice": 0.962924558734122,
+                "FN": 3181.836956521739,
+                "FP": 3034.2010869565215,
+                "IoU": 0.942225506773908,
+                "TN": 21548873.657608695,
+                "TP": 139959.30314009663,
+                "n_pred": 142993.50422705314,
+                "n_ref": 143141.14009661836
+            },
+            "7": {
+                "Dice": 0.9757046774007727,
+                "FN": 6546.338768115942,
+                "FP": 7616.440217391304,
+                "IoU": 0.9654407257987498,
+                "TN": 20837978.898550726,
+                "TP": 842907.3212560386,
+                "n_pred": 850523.76147343,
+                "n_ref": 849453.6600241546
+            }
+        }
+    },
+    "postprocessed": {
+        "foreground_mean": {
+            "Dice": 0.947831003281058,
+            "FN": 24221.409679089025,
+            "FP": 23231.02648378192,
+            "IoU": 0.9111595512012813,
+            "TN": 20772368.95954106,
+            "TP": 875227.6030883369,
+            "n_pred": 898458.6295721186,
+            "n_ref": 899449.0127674259
+        },
+        "mean": {
+            "1": {
+                "Dice": 0.9663381582297933,
+                "FN": 51781.09178743961,
+                "FP": 54967.756642512075,
+                "IoU": 0.9378369176697793,
+                "TN": 19720238.92753623,
+                "TP": 1868061.222826087,
+                "n_pred": 1923028.979468599,
+                "n_ref": 1919842.3146135265
+            },
+            "2": {
+                "Dice": 0.957884407011915,
+                "FN": 21668.210144927536,
+                "FP": 22863.38345410628,
+                "IoU": 0.9217993012824045,
+                "TN": 21059716.478864733,
+                "TP": 590800.9263285024,
+                "n_pred": 613664.3097826086,
+                "n_ref": 612469.13647343
+            },
+            "3": {
+                "Dice": 0.9672303238506204,
+                "FN": 45615.25,
+                "FP": 36531.213768115944,
+                "IoU": 0.940791967283391,
+                "TN": 19712272.719806764,
+                "TP": 1900629.8152173914,
+                "n_pred": 1937161.0289855073,
+                "n_ref": 1946245.0652173914
+            },
+            "4": {
+                "Dice": 0.9271283978454077,
+                "FN": 31290.937198067633,
+                "FP": 29929.147946859903,
+                "IoU": 0.8753949303608027,
+                "TN": 20927857.83574879,
+                "TP": 705971.0778985508,
+                "n_pred": 735900.2258454106,
+                "n_ref": 737262.0150966183
+            },
+            "5": {
+                "Dice": 0.8775996178802443,
+                "FN": 9464.621376811594,
+                "FP": 7677.018719806763,
+                "IoU": 0.7946187000311267,
+                "TN": 21599642.222222224,
+                "TP": 78265.13647342996,
+                "n_pred": 85942.15519323671,
+                "n_ref": 87729.75785024154
+            },
+            "6": {
+                "Dice": 0.9629314407486529,
+                "FN": 3183.4184782608695,
+                "FP": 3032.2246376811595,
+                "IoU": 0.9422343159827147,
+                "TN": 21548875.634057973,
+                "TP": 139957.7216183575,
+                "n_pred": 142989.94625603865,
+                "n_ref": 143141.14009661836
+            },
+            "7": {
+                "Dice": 0.9757046774007727,
+                "FN": 6546.338768115942,
+                "FP": 7616.440217391304,
+                "IoU": 0.9654407257987498,
+                "TN": 20837978.898550726,
+                "TP": 842907.3212560386,
+                "n_pred": 850523.76147343,
+                "n_ref": 849453.6600241546
+            }
+        }
+    },
+    "postprocessing_fns": [
+        "remove_all_but_largest_component_from_segmentation"
+    ],
+    "postprocessing_kwargs": [
+        {
+            "labels_or_regions": 6
+        }
+    ]
+}

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/crossval_results_folds_0_1_2_3_4/postprocessing.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:03db52a2a175b6c46f0f8ba75f3d969a3d33e8f567ced072cfa4bc1e8715a82b
+size 256

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/dataset.json

@@ -0,0 +1,22 @@
+{
+  "channel_names": {
+    "0": "CT"
+  },
+  "labels": {
+    "background": 0,
+    "muscle": 1,
+    "bone": 2,
+    "subcutaneous": 3,
+    "abdominal": 4,
+    "thoracic": 5,
+    "heart": 6,
+    "lungs": 7
+  },
+  "numTraining": 1656,
+  "file_ending": ".nii.gz",
+  "dataset_name": "BodyCompositionCT Compartment Labels v1",
+  "reference": "https://doi.org/10.1038/s41598-025-96238-6; https://doi.org/10.1038/s41597-024-03337-6",
+  "release": "2024-08-29",
+  "license": "CC BY 4.0",
+  "description": "BodyCompositionCT segmentation labels. Labels 1, 3, 4, and 5 denote anatomical compartments; labels 2, 6, and 7 denote bone, heart, and lungs. The Abdominal and Thoracic compartments exclude organs and larger vessels. IMAT is not a separate output label. Training CTs originated from the SAROS and TotalSegmentator datasets and are not included."
+}

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/checkpoint_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d4eb90614fef10c2eb9aad574d4cb3517511d224f2f2da897fe4e4618513cd77
+size 819931462

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/debug.json

@@ -0,0 +1,53 @@
+{
+    "_best_ema": "0.9626671837823639",
+    "batch_size": "2",
+    "configuration_manager": "{'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False}",
+    "configuration_name": "3d_fullres",
+    "cudnn_version": 8902,
+    "current_epoch": "1000",
+    "dataloader_train": "<nnunetv2.training.data_augmentation.custom_transforms.limited_length_multithreaded_augmenter.LimitedLenWrapper object at 0x7fefe563b0d0>",
+    "dataloader_train.generator": "<nnunetv2.training.dataloading.data_loader_3d.nnUNetDataLoader3D object at 0x7fefe563b3a0>",
+    "dataloader_train.num_processes": "16",
+    "dataloader_train.transform": "Compose ( [SpatialTransform( independent_scale_for_each_axis = False, p_rot_per_sample = 0.2, p_scale_per_sample = 0.2, p_el_per_sample = 0, data_key = 'data', label_key = 'seg', patch_size = [192, 192, 192], patch_center_dist_from_border = None, do_elastic_deform = False, alpha = (0, 0), sigma = (0, 0), do_rotation = True, angle_x = (-0.5235987755982988, 0.5235987755982988), angle_y = (-0.5235987755982988, 0.5235987755982988), angle_z = (-0.5235987755982988, 0.5235987755982988), do_scale = True, scale = (0.7, 1.4), border_mode_data = 'constant', border_cval_data = 0, order_data = 3, border_mode_seg = 'constant', border_cval_seg = -1, order_seg = 1, random_crop = False, p_rot_per_axis = 1, p_independent_scale_per_axis = 1 ), GaussianNoiseTransform( p_per_sample = 0.1, data_key = 'data', noise_variance = (0, 0.1), p_per_channel = 1, per_channel = False ), GaussianBlurTransform( p_per_sample = 0.2, different_sigma_per_channel = True, p_per_channel = 0.5, data_key = 'data', blur_sigma = (0.5, 1.0), different_sigma_per_axis = False, p_isotropic = 0 ), BrightnessMultiplicativeTransform( p_per_sample = 0.15, data_key = 'data', multiplier_range = (0.75, 1.25), per_channel = True ), ContrastAugmentationTransform( p_per_sample = 0.15, data_key = 'data', contrast_range = (0.75, 1.25), preserve_range = True, per_channel = True, p_per_channel = 1 ), SimulateLowResolutionTransform( order_upsample = 3, order_downsample = 0, channels = None, per_channel = True, p_per_channel = 0.5, p_per_sample = 0.25, data_key = 'data', zoom_range = (0.5, 1), ignore_axes = None ), GammaTransform( p_per_sample = 0.1, retain_stats = True, per_channel = True, data_key = 'data', gamma_range = (0.7, 1.5), invert_image = True ), GammaTransform( p_per_sample = 0.3, retain_stats = True, per_channel = True, data_key = 'data', gamma_range = (0.7, 1.5), invert_image = False ), MirrorTransform( p_per_sample = 1, data_key = 'data', label_key = 'seg', axes = (0, 1, 2) ), RemoveLabelTransform( output_key = 'seg', input_key = 'seg', replace_with = 0, remove_label = -1 ), RenameTransform( delete_old = True, out_key = 'target', in_key = 'seg' ), DownsampleSegForDSTransform2( axes = None, output_key = 'target', input_key = 'target', order = 0, ds_scales = [[1.0, 1.0, 1.0], [0.5, 0.5, 0.5], [0.25, 0.25, 0.25], [0.125, 0.125, 0.125], [0.0625, 0.0625, 0.0625]] ), NumpyToTensor( keys = ['data', 'target'], cast_to = 'float' )] )",
+    "dataloader_val": "<nnunetv2.training.data_augmentation.custom_transforms.limited_length_multithreaded_augmenter.LimitedLenWrapper object at 0x7fefe563b040>",
+    "dataloader_val.generator": "<nnunetv2.training.dataloading.data_loader_3d.nnUNetDataLoader3D object at 0x7fefe563b220>",
+    "dataloader_val.num_processes": "8",
+    "dataloader_val.transform": "Compose ( [RemoveLabelTransform( output_key = 'seg', input_key = 'seg', replace_with = 0, remove_label = -1 ), RenameTransform( delete_old = True, out_key = 'target', in_key = 'seg' ), DownsampleSegForDSTransform2( axes = None, output_key = 'target', input_key = 'target', order = 0, ds_scales = [[1.0, 1.0, 1.0], [0.5, 0.5, 0.5], [0.25, 0.25, 0.25], [0.125, 0.125, 0.125], [0.0625, 0.0625, 0.0625]] ), NumpyToTensor( keys = ['data', 'target'], cast_to = 'float' )] )",
+    "dataset_json": "{'channel_names': {'0': 'CT'}, 'labels': {'background': 0, 'muscle': 1, 'bone': 2, 'subcutaneous': 3, 'abdominal': 4, 'thoracic': 5, 'heart': 6, 'lungs': 7}, 'numTraining': 1656, 'file_ending': '.nii.gz', 'dataset_name': 'BodyCompositionCT Compartment Labels v1', 'reference': 'https://doi.org/10.1038/s41598-025-96238-6; https://doi.org/10.1038/s41597-024-03337-6', 'release': '2024-08-29', 'license': 'CC BY 4.0', 'description': 'BodyCompositionCT segmentation labels. Labels 1, 3, 4, and 5 denote anatomical compartments; labels 2, 6, and 7 denote bone, heart, and lungs. The Abdominal and Thoracic compartments exclude organs and larger vessels. IMAT is not a separate output label. Training CTs originated from the SAROS and TotalSegmentator datasets and are not included.'}",
+    "device": "cuda:0",
+    "disable_checkpointing": "False",
+    "enable_deep_supervision": "True",
+    "fold": "0",
+    "folder_with_segs_from_previous_stage": "None",
+    "gpu_name": "NVIDIA A100-SXM4-40GB",
+    "grad_scaler": "<torch.cuda.amp.grad_scaler.GradScaler object at 0x7ff097c6b550>",
+    "hostname": "<training-host>",
+    "inference_allowed_mirroring_axes": "(0, 1, 2)",
+    "initial_lr": "0.01",
+    "is_cascaded": "False",
+    "is_ddp": "False",
+    "label_manager": "<nnunetv2.utilities.label_handling.label_handling.LabelManager object at 0x7fef71116ec0>",
+    "local_rank": "0",
+    "log_file": "<workspace>/data/nnUNet_results/Dataset611_Tissue/nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/training_log_2024_9_11_14_42_12.txt",
+    "logger": "<nnunetv2.training.logging.nnunet_logger.nnUNetLogger object at 0x7fef70bb35b0>",
+    "loss": "DeepSupervisionWrapper(\n  (loss): DC_and_CE_loss(\n    (ce): RobustCrossEntropyLoss()\n    (dc): MemoryEfficientSoftDiceLoss()\n  )\n)",
+    "lr_scheduler": "<nnunetv2.training.lr_scheduler.polylr.PolyLRScheduler object at 0x7fef71831270>",
+    "my_init_kwargs": "{'plans': {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'configurations': {'2d': {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 89, 'patch_size': [320, 320], 'median_image_size_in_voxels': [257.0, 260.0], 'spacing': [1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 7, 'features_per_stage': [32, 64, 128, 256, 512, 512, 512], 'conv_op': 'torch.nn.modules.conv.Conv2d', 'kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'strides': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm2d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': True}, '3d_fullres': {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False}}, 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}}, 'configuration': '3d_fullres', 'fold': 0, 'dataset_json': {'channel_names': {'0': 'CT'}, 'labels': {'background': 0, 'muscle': 1, 'bone': 2, 'subcutaneous': 3, 'abdominal': 4, 'thoracic': 5, 'heart': 6, 'lungs': 7}, 'numTraining': 1656, 'file_ending': '.nii.gz', 'dataset_name': 'BodyCompositionCT Compartment Labels v1', 'reference': 'https://doi.org/10.1038/s41598-025-96238-6; https://doi.org/10.1038/s41597-024-03337-6', 'release': '2024-08-29', 'license': 'CC BY 4.0', 'description': 'BodyCompositionCT segmentation labels. Labels 1, 3, 4, and 5 denote anatomical compartments; labels 2, 6, and 7 denote bone, heart, and lungs. The Abdominal and Thoracic compartments exclude organs and larger vessels. IMAT is not a separate output label. Training CTs originated from the SAROS and TotalSegmentator datasets and are not included.'}, 'unpack_dataset': False, 'device': device(type='cuda')}",
+    "network": "OptimizedModule",
+    "num_epochs": "1000",
+    "num_input_channels": "1",
+    "num_iterations_per_epoch": "250",
+    "num_val_iterations_per_epoch": "50",
+    "optimizer": "SGD (\nParameter Group 0\n    dampening: 0\n    differentiable: False\n    foreach: None\n    fused: None\n    initial_lr: 0.01\n    lr: 1.995262314968881e-05\n    maximize: False\n    momentum: 0.99\n    nesterov: True\n    weight_decay: 3e-05\n)",
+    "output_folder": "<workspace>/data/nnUNet_results/Dataset611_Tissue/nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0",
+    "output_folder_base": "<workspace>/data/nnUNet_results/Dataset611_Tissue/nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres",
+    "oversample_foreground_percent": "0.33",
+    "plans_manager": "{'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'configurations': {'2d': {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 89, 'patch_size': [320, 320], 'median_image_size_in_voxels': [257.0, 260.0], 'spacing': [1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 7, 'features_per_stage': [32, 64, 128, 256, 512, 512, 512], 'conv_op': 'torch.nn.modules.conv.Conv2d', 'kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'strides': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm2d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': True}, '3d_fullres': {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False}}, 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}}",
+    "preprocessed_dataset_folder": "<workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/nnUNetPlans_3d_fullres",
+    "preprocessed_dataset_folder_base": "<workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue",
+    "save_every": "50",
+    "torch_version": "2.3.0+cu121",
+    "unpack_dataset": "False",
+    "was_initialized": "True",
+    "weight_decay": "3e-05"
+}

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_10_06_01_54.txt

@@ -0,0 +1,371 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-10 06:01:57.850573: Using torch.compile... 
+2024-09-10 06:02:04.490486: do_dummy_2d_data_aug: False 
+2024-09-10 06:02:04.496485: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-10 06:02:04.498019: The split file contains 5 splits. 
+2024-09-10 06:02:04.498554: Desired fold for training: 0 
+2024-09-10 06:02:04.499049: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-10 06:02:20.093287: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-10 06:02:22.551358:  
+2024-09-10 06:02:22.572980: Epoch 700 
+2024-09-10 06:02:22.574784: Current learning rate: 0.00338 
+2024-09-10 06:12:33.532906: train_loss -0.7484 
+2024-09-10 06:12:33.535982: val_loss -0.7562 
+2024-09-10 06:12:33.536785: Pseudo dice [0.9656, 0.9596, 0.9797, 0.9479, 0.8752, 0.9722, 0.9912] 
+2024-09-10 06:12:33.537721: Epoch time: 610.99 s 
+2024-09-10 06:12:35.774765:  
+2024-09-10 06:12:35.775748: Epoch 701 
+2024-09-10 06:12:35.776415: Current learning rate: 0.00337 
+2024-09-10 06:18:03.709767: train_loss -0.7485 
+2024-09-10 06:18:03.725631: val_loss -0.7711 
+2024-09-10 06:18:03.726658: Pseudo dice [0.9649, 0.958, 0.9809, 0.9499, 0.8832, 0.974, 0.9914] 
+2024-09-10 06:18:03.728029: Epoch time: 327.94 s 
+2024-09-10 06:18:09.599058:  
+2024-09-10 06:18:09.612869: Epoch 702 
+2024-09-10 06:18:09.614774: Current learning rate: 0.00336 
+2024-09-10 06:23:58.681911: train_loss -0.7579 
+2024-09-10 06:23:58.697533: val_loss -0.7459 
+2024-09-10 06:23:58.698497: Pseudo dice [0.9634, 0.9575, 0.9772, 0.952, 0.8776, 0.9721, 0.9891] 
+2024-09-10 06:23:58.699421: Epoch time: 349.09 s 
+2024-09-10 06:24:03.894700:  
+2024-09-10 06:24:03.908867: Epoch 703 
+2024-09-10 06:24:03.909787: Current learning rate: 0.00335 
+2024-09-10 06:29:45.790906: train_loss -0.7545 
+2024-09-10 06:29:45.808509: val_loss -0.762 
+2024-09-10 06:29:45.811442: Pseudo dice [0.9682, 0.9607, 0.9823, 0.9474, 0.8731, 0.9717, 0.9916] 
+2024-09-10 06:29:45.814188: Epoch time: 341.9 s 
+2024-09-10 06:29:50.702482:  
+2024-09-10 06:29:50.704561: Epoch 704 
+2024-09-10 06:29:50.705794: Current learning rate: 0.00334 
+2024-09-10 06:35:24.447131: train_loss -0.7444 
+2024-09-10 06:35:24.450030: val_loss -0.7661 
+2024-09-10 06:35:24.450981: Pseudo dice [0.9661, 0.9596, 0.9792, 0.9515, 0.8796, 0.9718, 0.9914] 
+2024-09-10 06:35:24.451738: Epoch time: 333.76 s 
+2024-09-10 06:35:28.289249:  
+2024-09-10 06:35:28.290351: Epoch 705 
+2024-09-10 06:35:28.291122: Current learning rate: 0.00333 
+2024-09-10 06:40:52.697942: train_loss -0.752 
+2024-09-10 06:40:52.700243: val_loss -0.739 
+2024-09-10 06:40:52.701034: Pseudo dice [0.9643, 0.9576, 0.9799, 0.9438, 0.8814, 0.9707, 0.9909] 
+2024-09-10 06:40:52.701737: Epoch time: 324.41 s 
+2024-09-10 06:40:56.353730:  
+2024-09-10 06:40:56.368740: Epoch 706 
+2024-09-10 06:40:56.369806: Current learning rate: 0.00332 
+2024-09-10 06:46:58.570993: train_loss -0.749 
+2024-09-10 06:46:58.591032: val_loss -0.7939 
+2024-09-10 06:46:58.592181: Pseudo dice [0.967, 0.961, 0.9824, 0.9569, 0.8822, 0.9701, 0.9905] 
+2024-09-10 06:46:58.592841: Epoch time: 362.22 s 
+2024-09-10 06:47:03.359958:  
+2024-09-10 06:47:03.361192: Epoch 707 
+2024-09-10 06:47:03.362050: Current learning rate: 0.00331 
+2024-09-10 06:52:42.934469: train_loss -0.7294 
+2024-09-10 06:52:42.936774: val_loss -0.738 
+2024-09-10 06:52:42.937844: Pseudo dice [0.9648, 0.9556, 0.9792, 0.9464, 0.8805, 0.9704, 0.9904] 
+2024-09-10 06:52:42.938608: Epoch time: 339.58 s 
+2024-09-10 06:52:48.532223:  
+2024-09-10 06:52:48.549480: Epoch 708 
+2024-09-10 06:52:48.550958: Current learning rate: 0.0033 
+2024-09-10 06:58:22.803580: train_loss -0.7491 
+2024-09-10 06:58:22.816117: val_loss -0.7796 
+2024-09-10 06:58:22.817590: Pseudo dice [0.967, 0.9606, 0.9798, 0.9535, 0.8809, 0.9741, 0.9914] 
+2024-09-10 06:58:22.818572: Epoch time: 334.28 s 
+2024-09-10 06:58:25.372842:  
+2024-09-10 06:58:25.388440: Epoch 709 
+2024-09-10 06:58:25.389801: Current learning rate: 0.00329 
+2024-09-10 07:04:09.460056: train_loss -0.7273 
+2024-09-10 07:04:09.479719: val_loss -0.7658 
+2024-09-10 07:04:09.481483: Pseudo dice [0.9601, 0.9546, 0.9752, 0.9499, 0.8746, 0.9726, 0.9899] 
+2024-09-10 07:04:09.483304: Epoch time: 344.09 s 
+2024-09-10 07:04:14.614452:  
+2024-09-10 07:04:14.615935: Epoch 710 
+2024-09-10 07:04:14.616790: Current learning rate: 0.00328 
+2024-09-10 07:10:19.316053: train_loss -0.7427 
+2024-09-10 07:10:19.326909: val_loss -0.7297 
+2024-09-10 07:10:19.328372: Pseudo dice [0.9636, 0.9605, 0.9772, 0.9499, 0.8697, 0.9647, 0.9888] 
+2024-09-10 07:10:19.329490: Epoch time: 364.71 s 
+2024-09-10 07:10:27.098127:  
+2024-09-10 07:10:27.100003: Epoch 711 
+2024-09-10 07:10:27.101314: Current learning rate: 0.00327 
+2024-09-10 07:15:53.554344: train_loss -0.732 
+2024-09-10 07:15:53.569824: val_loss -0.7547 
+2024-09-10 07:15:53.570755: Pseudo dice [0.9587, 0.9563, 0.9739, 0.9493, 0.8707, 0.971, 0.9886] 
+2024-09-10 07:15:53.571505: Epoch time: 326.46 s 
+2024-09-10 07:15:58.310495:  
+2024-09-10 07:15:58.323893: Epoch 712 
+2024-09-10 07:15:58.325227: Current learning rate: 0.00326 
+2024-09-10 07:21:28.172969: train_loss -0.7479 
+2024-09-10 07:21:28.180860: val_loss -0.7564 
+2024-09-10 07:21:28.182811: Pseudo dice [0.9645, 0.9576, 0.9782, 0.9466, 0.8876, 0.9729, 0.9923] 
+2024-09-10 07:21:28.195731: Epoch time: 329.88 s 
+2024-09-10 07:21:32.233674:  
+2024-09-10 07:21:32.234771: Epoch 713 
+2024-09-10 07:21:32.235443: Current learning rate: 0.00325 
+2024-09-10 07:26:55.375213: train_loss -0.7341 
+2024-09-10 07:26:55.387078: val_loss -0.763 
+2024-09-10 07:26:55.388745: Pseudo dice [0.9685, 0.9594, 0.982, 0.9513, 0.881, 0.9723, 0.9916] 
+2024-09-10 07:26:55.390136: Epoch time: 323.15 s 
+2024-09-10 07:26:59.677455:  
+2024-09-10 07:26:59.678836: Epoch 714 
+2024-09-10 07:26:59.679956: Current learning rate: 0.00324 
+2024-09-10 07:32:04.013855: train_loss -0.7418 
+2024-09-10 07:32:04.030016: val_loss -0.7405 
+2024-09-10 07:32:04.031161: Pseudo dice [0.9665, 0.9599, 0.9812, 0.9506, 0.8826, 0.9729, 0.992] 
+2024-09-10 07:32:04.032031: Epoch time: 304.34 s 
+2024-09-10 07:32:07.416206:  
+2024-09-10 07:32:07.424847: Epoch 715 
+2024-09-10 07:32:07.431886: Current learning rate: 0.00323 
+2024-09-10 07:37:14.992293: train_loss -0.7523 
+2024-09-10 07:37:15.007690: val_loss -0.7572 
+2024-09-10 07:37:15.008777: Pseudo dice [0.9661, 0.9589, 0.9811, 0.9514, 0.8876, 0.9737, 0.9924] 
+2024-09-10 07:37:15.009937: Epoch time: 307.58 s 
+2024-09-10 07:37:18.638755:  
+2024-09-10 07:37:18.639966: Epoch 716 
+2024-09-10 07:37:18.640823: Current learning rate: 0.00322 
+2024-09-10 07:42:37.825489: train_loss -0.7722 
+2024-09-10 07:42:37.837811: val_loss -0.7525 
+2024-09-10 07:42:37.838633: Pseudo dice [0.96, 0.9569, 0.9779, 0.9473, 0.8825, 0.9681, 0.9901] 
+2024-09-10 07:42:37.839351: Epoch time: 319.19 s 
+2024-09-10 07:42:41.588879:  
+2024-09-10 07:42:41.590043: Epoch 717 
+2024-09-10 07:42:41.591161: Current learning rate: 0.00321 
+2024-09-10 07:47:59.035605: train_loss -0.7572 
+2024-09-10 07:47:59.037019: val_loss -0.7439 
+2024-09-10 07:47:59.038152: Pseudo dice [0.9642, 0.96, 0.9751, 0.9526, 0.8782, 0.9715, 0.9904] 
+2024-09-10 07:47:59.038947: Epoch time: 317.45 s 
+2024-09-10 07:48:02.149020:  
+2024-09-10 07:48:02.162264: Epoch 718 
+2024-09-10 07:48:02.163433: Current learning rate: 0.0032 
+2024-09-10 07:53:14.334403: train_loss -0.7426 
+2024-09-10 07:53:14.349552: val_loss -0.7562 
+2024-09-10 07:53:14.350237: Pseudo dice [0.967, 0.9589, 0.978, 0.9514, 0.8761, 0.971, 0.991] 
+2024-09-10 07:53:14.350884: Epoch time: 312.19 s 
+2024-09-10 07:53:16.818849:  
+2024-09-10 07:53:16.820216: Epoch 719 
+2024-09-10 07:53:16.821135: Current learning rate: 0.00319 
+2024-09-10 07:58:35.380176: train_loss -0.7331 
+2024-09-10 07:58:35.382705: val_loss -0.7707 
+2024-09-10 07:58:35.383569: Pseudo dice [0.9668, 0.9594, 0.9811, 0.9508, 0.889, 0.9728, 0.9917] 
+2024-09-10 07:58:35.384311: Epoch time: 318.57 s 
+2024-09-10 07:58:39.596968:  
+2024-09-10 07:58:39.597984: Epoch 720 
+2024-09-10 07:58:39.598633: Current learning rate: 0.00318 
+2024-09-10 08:04:25.452646: train_loss -0.7681 
+2024-09-10 08:04:25.476781: val_loss -0.7379 
+2024-09-10 08:04:25.477633: Pseudo dice [0.965, 0.9593, 0.9783, 0.9457, 0.8902, 0.9734, 0.9918] 
+2024-09-10 08:04:25.478295: Epoch time: 345.86 s 
+2024-09-10 08:04:29.159787:  
+2024-09-10 08:04:29.160789: Epoch 721 
+2024-09-10 08:04:29.161513: Current learning rate: 0.00317 
+2024-09-10 08:09:54.951123: train_loss -0.7452 
+2024-09-10 08:09:54.965280: val_loss -0.7165 
+2024-09-10 08:09:54.966173: Pseudo dice [0.9663, 0.9598, 0.9787, 0.9524, 0.8033, 0.9686, 0.9696] 
+2024-09-10 08:09:54.967025: Epoch time: 325.8 s 
+2024-09-10 08:09:59.020332:  
+2024-09-10 08:09:59.030329: Epoch 722 
+2024-09-10 08:09:59.031564: Current learning rate: 0.00316 
+2024-09-10 08:15:17.747986: train_loss -0.7489 
+2024-09-10 08:15:17.762274: val_loss -0.7454 
+2024-09-10 08:15:17.763060: Pseudo dice [0.9635, 0.9575, 0.9814, 0.9432, 0.8788, 0.9692, 0.9899] 
+2024-09-10 08:15:17.764153: Epoch time: 318.74 s 
+2024-09-10 08:15:21.646237:  
+2024-09-10 08:15:21.660679: Epoch 723 
+2024-09-10 08:15:21.662282: Current learning rate: 0.00315 
+2024-09-10 08:20:41.083967: train_loss -0.7433 
+2024-09-10 08:20:41.100617: val_loss -0.7172 
+2024-09-10 08:20:41.101437: Pseudo dice [0.9628, 0.9569, 0.9786, 0.9505, 0.8767, 0.9692, 0.9921] 
+2024-09-10 08:20:41.102371: Epoch time: 319.44 s 
+2024-09-10 08:20:46.548774:  
+2024-09-10 08:20:46.549780: Epoch 724 
+2024-09-10 08:20:46.550503: Current learning rate: 0.00314 
+2024-09-10 08:26:05.036397: train_loss -0.7319 
+2024-09-10 08:26:05.084071: val_loss -0.7578 
+2024-09-10 08:26:05.085435: Pseudo dice [0.9634, 0.9564, 0.9777, 0.946, 0.8752, 0.9675, 0.9905] 
+2024-09-10 08:26:05.086814: Epoch time: 318.49 s 
+2024-09-10 08:26:08.890269:  
+2024-09-10 08:26:08.904677: Epoch 725 
+2024-09-10 08:26:08.906828: Current learning rate: 0.00313 
+2024-09-10 08:31:04.922113: train_loss -0.7381 
+2024-09-10 08:31:04.945651: val_loss -0.7628 
+2024-09-10 08:31:04.947149: Pseudo dice [0.9654, 0.9579, 0.9812, 0.9517, 0.887, 0.9721, 0.9919] 
+2024-09-10 08:31:04.948715: Epoch time: 296.04 s 
+2024-09-10 08:31:08.459878:  
+2024-09-10 08:31:08.472639: Epoch 726 
+2024-09-10 08:31:08.473903: Current learning rate: 0.00312 
+2024-09-10 08:36:00.379124: train_loss -0.7466 
+2024-09-10 08:36:00.381635: val_loss -0.7574 
+2024-09-10 08:36:00.382702: Pseudo dice [0.9613, 0.9565, 0.9759, 0.949, 0.8737, 0.9685, 0.9908] 
+2024-09-10 08:36:00.383642: Epoch time: 291.92 s 
+2024-09-10 08:36:03.692892:  
+2024-09-10 08:36:03.693839: Epoch 727 
+2024-09-10 08:36:03.694498: Current learning rate: 0.00311 
+2024-09-10 08:40:51.811333: train_loss -0.7511 
+2024-09-10 08:40:51.818222: val_loss -0.747 
+2024-09-10 08:40:51.832871: Pseudo dice [0.9641, 0.9595, 0.9788, 0.9495, 0.8694, 0.9713, 0.982] 
+2024-09-10 08:40:51.839284: Epoch time: 288.12 s 
+2024-09-10 08:40:55.281301:  
+2024-09-10 08:40:55.282167: Epoch 728 
+2024-09-10 08:40:55.282895: Current learning rate: 0.0031 
+2024-09-10 08:46:00.566040: train_loss -0.7397 
+2024-09-10 08:46:00.567489: val_loss -0.7667 
+2024-09-10 08:46:00.568313: Pseudo dice [0.9695, 0.9614, 0.9829, 0.9547, 0.8818, 0.9655, 0.9908] 
+2024-09-10 08:46:00.569346: Epoch time: 305.29 s 
+2024-09-10 08:46:03.354722:  
+2024-09-10 08:46:03.355724: Epoch 729 
+2024-09-10 08:46:03.356523: Current learning rate: 0.00309 
+2024-09-10 08:51:01.292442: train_loss -0.7402 
+2024-09-10 08:51:01.308835: val_loss -0.7658 
+2024-09-10 08:51:01.309609: Pseudo dice [0.9638, 0.9571, 0.9782, 0.9493, 0.886, 0.9728, 0.991] 
+2024-09-10 08:51:01.310486: Epoch time: 297.94 s 
+2024-09-10 08:51:04.567293:  
+2024-09-10 08:51:04.568478: Epoch 730 
+2024-09-10 08:51:04.569283: Current learning rate: 0.00308 
+2024-09-10 08:55:43.892802: train_loss -0.7424 
+2024-09-10 08:55:43.909172: val_loss -0.7567 
+2024-09-10 08:55:43.910249: Pseudo dice [0.9651, 0.9586, 0.9767, 0.9476, 0.8795, 0.9727, 0.9915] 
+2024-09-10 08:55:43.911278: Epoch time: 279.33 s 
+2024-09-10 08:55:47.696615:  
+2024-09-10 08:55:47.697651: Epoch 731 
+2024-09-10 08:55:47.698334: Current learning rate: 0.00307 
+2024-09-10 09:00:53.539560: train_loss -0.7437 
+2024-09-10 09:00:53.541253: val_loss -0.7561 
+2024-09-10 09:00:53.542266: Pseudo dice [0.9662, 0.9587, 0.9788, 0.9509, 0.8868, 0.9713, 0.9911] 
+2024-09-10 09:00:53.543218: Epoch time: 305.85 s 
+2024-09-10 09:00:57.372066:  
+2024-09-10 09:00:57.389518: Epoch 732 
+2024-09-10 09:00:57.390610: Current learning rate: 0.00306 
+2024-09-10 09:06:02.895200: train_loss -0.719 
+2024-09-10 09:06:02.908918: val_loss -0.7585 
+2024-09-10 09:06:02.909700: Pseudo dice [0.9664, 0.9591, 0.9805, 0.9559, 0.8857, 0.9706, 0.9917] 
+2024-09-10 09:06:02.910350: Epoch time: 305.54 s 
+2024-09-10 09:06:07.837104:  
+2024-09-10 09:06:07.838431: Epoch 733 
+2024-09-10 09:06:07.839212: Current learning rate: 0.00305 
+2024-09-10 09:11:34.247800: train_loss -0.7351 
+2024-09-10 09:11:34.272656: val_loss -0.738 
+2024-09-10 09:11:34.273517: Pseudo dice [0.9676, 0.9616, 0.9832, 0.9532, 0.8714, 0.9714, 0.9889] 
+2024-09-10 09:11:34.274329: Epoch time: 326.42 s 
+2024-09-10 09:11:37.012565:  
+2024-09-10 09:11:37.022403: Epoch 734 
+2024-09-10 09:11:37.023259: Current learning rate: 0.00304 
+2024-09-10 09:16:46.848454: train_loss -0.7429 
+2024-09-10 09:16:46.865039: val_loss -0.7571 
+2024-09-10 09:16:46.867296: Pseudo dice [0.9661, 0.9568, 0.9807, 0.9543, 0.8851, 0.9721, 0.9902] 
+2024-09-10 09:16:46.869032: Epoch time: 309.84 s 
+2024-09-10 09:16:50.432327:  
+2024-09-10 09:16:50.433357: Epoch 735 
+2024-09-10 09:16:50.434065: Current learning rate: 0.00303 
+2024-09-10 09:22:15.423707: train_loss -0.7588 
+2024-09-10 09:22:15.428261: val_loss -0.7234 
+2024-09-10 09:22:15.429213: Pseudo dice [0.9677, 0.9611, 0.9821, 0.9532, 0.8726, 0.9706, 0.9891] 
+2024-09-10 09:22:15.430222: Epoch time: 325.0 s 
+2024-09-10 09:22:19.165472:  
+2024-09-10 09:22:19.166869: Epoch 736 
+2024-09-10 09:22:19.167741: Current learning rate: 0.00302 
+2024-09-10 09:27:46.251540: train_loss -0.7481 
+2024-09-10 09:27:46.270417: val_loss -0.7722 
+2024-09-10 09:27:46.271682: Pseudo dice [0.9653, 0.9608, 0.9809, 0.9574, 0.8943, 0.9743, 0.992] 
+2024-09-10 09:27:46.272621: Epoch time: 327.09 s 
+2024-09-10 09:27:50.206029:  
+2024-09-10 09:27:50.220505: Epoch 737 
+2024-09-10 09:27:50.221392: Current learning rate: 0.00301 
+2024-09-10 09:33:15.823102: train_loss -0.7539 
+2024-09-10 09:33:15.834583: val_loss -0.7305 
+2024-09-10 09:33:15.836233: Pseudo dice [0.9667, 0.9608, 0.977, 0.9566, 0.8834, 0.9714, 0.9894] 
+2024-09-10 09:33:15.837576: Epoch time: 325.62 s 
+2024-09-10 09:33:20.984639:  
+2024-09-10 09:33:20.986485: Epoch 738 
+2024-09-10 09:33:21.004009: Current learning rate: 0.003 
+2024-09-10 09:38:25.179481: train_loss -0.7527 
+2024-09-10 09:38:25.201435: val_loss -0.7671 
+2024-09-10 09:38:25.202555: Pseudo dice [0.9634, 0.9569, 0.9777, 0.9412, 0.8806, 0.9737, 0.9918] 
+2024-09-10 09:38:25.203305: Epoch time: 304.21 s 
+2024-09-10 09:38:28.911854:  
+2024-09-10 09:38:28.924540: Epoch 739 
+2024-09-10 09:38:28.925508: Current learning rate: 0.00299 
+2024-09-10 09:43:58.082158: train_loss -0.7545 
+2024-09-10 09:43:58.088454: val_loss -0.7454 
+2024-09-10 09:43:58.089738: Pseudo dice [0.9662, 0.962, 0.9798, 0.9509, 0.8754, 0.9728, 0.9907] 
+2024-09-10 09:43:58.090609: Epoch time: 329.18 s 
+2024-09-10 09:44:02.371006:  
+2024-09-10 09:44:02.373269: Epoch 740 
+2024-09-10 09:44:02.375094: Current learning rate: 0.00297 
+2024-09-10 09:49:29.716605: train_loss -0.7537 
+2024-09-10 09:49:29.718259: val_loss -0.7115 
+2024-09-10 09:49:29.719058: Pseudo dice [0.9659, 0.9605, 0.978, 0.951, 0.883, 0.9705, 0.9913] 
+2024-09-10 09:49:29.719823: Epoch time: 327.36 s 
+2024-09-10 09:49:32.486867:  
+2024-09-10 09:49:32.490034: Epoch 741 
+2024-09-10 09:49:32.496768: Current learning rate: 0.00296 
+2024-09-10 09:54:51.388134: train_loss -0.7562 
+2024-09-10 09:54:51.406152: val_loss -0.7523 
+2024-09-10 09:54:51.406888: Pseudo dice [0.9677, 0.9589, 0.9792, 0.9502, 0.8872, 0.9745, 0.9922] 
+2024-09-10 09:54:51.407562: Epoch time: 318.91 s 
+2024-09-10 09:54:55.793814:  
+2024-09-10 09:54:55.808591: Epoch 742 
+2024-09-10 09:54:55.809764: Current learning rate: 0.00295 
+2024-09-10 10:00:25.497274: train_loss -0.7472 
+2024-09-10 10:00:25.517614: val_loss -0.737 
+2024-09-10 10:00:25.518855: Pseudo dice [0.9652, 0.9594, 0.9794, 0.9523, 0.8834, 0.9746, 0.9911] 
+2024-09-10 10:00:25.520098: Epoch time: 329.71 s 
+2024-09-10 10:00:29.597307:  
+2024-09-10 10:00:29.608852: Epoch 743 
+2024-09-10 10:00:29.610221: Current learning rate: 0.00294 
+2024-09-10 10:05:50.179054: train_loss -0.7561 
+2024-09-10 10:05:50.181725: val_loss -0.7695 
+2024-09-10 10:05:50.184157: Pseudo dice [0.9676, 0.96, 0.9824, 0.9556, 0.882, 0.973, 0.9928] 
+2024-09-10 10:05:50.186323: Epoch time: 320.59 s 
+2024-09-10 10:05:55.154057:  
+2024-09-10 10:05:55.175898: Epoch 744 
+2024-09-10 10:05:55.177965: Current learning rate: 0.00293 
+2024-09-10 10:11:19.303221: train_loss -0.7486 
+2024-09-10 10:11:19.304680: val_loss -0.7663 
+2024-09-10 10:11:19.305425: Pseudo dice [0.9659, 0.9583, 0.9791, 0.9531, 0.8897, 0.9741, 0.9924] 
+2024-09-10 10:11:19.306101: Epoch time: 324.16 s 
+2024-09-10 10:11:23.056201:  
+2024-09-10 10:11:23.057628: Epoch 745 
+2024-09-10 10:11:23.059000: Current learning rate: 0.00292 
+2024-09-10 10:16:41.512507: train_loss -0.7543 
+2024-09-10 10:16:41.532231: val_loss -0.7629 
+2024-09-10 10:16:41.533408: Pseudo dice [0.9688, 0.9633, 0.9806, 0.9523, 0.8888, 0.9738, 0.9922] 
+2024-09-10 10:16:41.534184: Epoch time: 318.46 s 
+2024-09-10 10:16:45.727365:  
+2024-09-10 10:16:45.728715: Epoch 746 
+2024-09-10 10:16:45.729676: Current learning rate: 0.00291 
+2024-09-10 10:22:14.202525: train_loss -0.7457 
+2024-09-10 10:22:14.205240: val_loss -0.7796 
+2024-09-10 10:22:14.206892: Pseudo dice [0.9647, 0.9595, 0.98, 0.9471, 0.8847, 0.9759, 0.9917] 
+2024-09-10 10:22:14.208655: Epoch time: 328.48 s 
+2024-09-10 10:22:19.135295:  
+2024-09-10 10:22:19.163850: Epoch 747 
+2024-09-10 10:22:19.165286: Current learning rate: 0.0029 
+2024-09-10 10:27:38.663837: train_loss -0.7449 
+2024-09-10 10:27:38.677553: val_loss -0.7474 
+2024-09-10 10:27:38.678643: Pseudo dice [0.9659, 0.9593, 0.9826, 0.9592, 0.881, 0.9732, 0.9905] 
+2024-09-10 10:27:38.679459: Epoch time: 319.54 s 
+2024-09-10 10:27:42.680107:  
+2024-09-10 10:27:42.681638: Epoch 748 
+2024-09-10 10:27:42.682947: Current learning rate: 0.00289 
+2024-09-10 10:32:45.454682: train_loss -0.7523 
+2024-09-10 10:32:45.468880: val_loss -0.7331 
+2024-09-10 10:32:45.469691: Pseudo dice [0.9614, 0.9594, 0.9754, 0.9497, 0.8806, 0.9716, 0.9917] 
+2024-09-10 10:32:45.470315: Epoch time: 302.82 s 
+2024-09-10 10:32:49.081366:  
+2024-09-10 10:32:49.097005: Epoch 749 
+2024-09-10 10:32:49.097912: Current learning rate: 0.00288 
+2024-09-10 10:38:17.797346: train_loss -0.7382 
+2024-09-10 10:38:17.821672: val_loss -0.7485 
+2024-09-10 10:38:17.822992: Pseudo dice [0.9654, 0.9553, 0.981, 0.9525, 0.8844, 0.9755, 0.9907] 
+2024-09-10 10:38:17.824611: Epoch time: 328.72 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_10_14_01_21.txt

@@ -0,0 +1,381 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-10 14:01:25.334631: Using torch.compile... 
+2024-09-10 14:01:31.559520: do_dummy_2d_data_aug: False 
+2024-09-10 14:01:31.565278: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-10 14:01:31.566705: The split file contains 5 splits. 
+2024-09-10 14:01:31.567277: Desired fold for training: 0 
+2024-09-10 14:01:31.568035: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-10 14:01:46.896044: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-10 14:01:49.842394:  
+2024-09-10 14:01:49.856340: Epoch 750 
+2024-09-10 14:01:49.857303: Current learning rate: 0.00287 
+2024-09-10 14:12:22.665801: train_loss -0.7379 
+2024-09-10 14:12:22.668086: val_loss -0.7587 
+2024-09-10 14:12:22.669441: Pseudo dice [0.9642, 0.9569, 0.9787, 0.9491, 0.8768, 0.9699, 0.9907] 
+2024-09-10 14:12:22.670309: Epoch time: 632.83 s 
+2024-09-10 14:12:24.869202:  
+2024-09-10 14:12:24.870508: Epoch 751 
+2024-09-10 14:12:24.871398: Current learning rate: 0.00286 
+2024-09-10 14:18:03.701409: train_loss -0.7428 
+2024-09-10 14:18:03.717026: val_loss -0.7344 
+2024-09-10 14:18:03.717813: Pseudo dice [0.9661, 0.9596, 0.9802, 0.9483, 0.8836, 0.9738, 0.9924] 
+2024-09-10 14:18:03.718457: Epoch time: 338.84 s 
+2024-09-10 14:18:07.530936:  
+2024-09-10 14:18:07.546041: Epoch 752 
+2024-09-10 14:18:07.547215: Current learning rate: 0.00285 
+2024-09-10 14:23:15.292379: train_loss -0.7558 
+2024-09-10 14:23:15.317312: val_loss -0.7503 
+2024-09-10 14:23:15.318569: Pseudo dice [0.9636, 0.9587, 0.9782, 0.9465, 0.8858, 0.9721, 0.9901] 
+2024-09-10 14:23:15.319365: Epoch time: 307.77 s 
+2024-09-10 14:23:20.339790:  
+2024-09-10 14:23:20.355727: Epoch 753 
+2024-09-10 14:23:20.356681: Current learning rate: 0.00284 
+2024-09-10 14:28:46.548770: train_loss -0.7498 
+2024-09-10 14:28:46.550328: val_loss -0.7571 
+2024-09-10 14:28:46.551145: Pseudo dice [0.9681, 0.962, 0.9835, 0.952, 0.8789, 0.9719, 0.9915] 
+2024-09-10 14:28:46.552081: Epoch time: 326.21 s 
+2024-09-10 14:28:49.122567:  
+2024-09-10 14:28:49.123538: Epoch 754 
+2024-09-10 14:28:49.124275: Current learning rate: 0.00283 
+2024-09-10 14:34:19.810613: train_loss -0.7658 
+2024-09-10 14:34:19.812106: val_loss -0.735 
+2024-09-10 14:34:19.812976: Pseudo dice [0.9624, 0.9614, 0.9771, 0.9487, 0.8725, 0.9724, 0.989] 
+2024-09-10 14:34:19.813709: Epoch time: 330.69 s 
+2024-09-10 14:34:23.329513:  
+2024-09-10 14:34:23.333582: Epoch 755 
+2024-09-10 14:34:23.334498: Current learning rate: 0.00282 
+2024-09-10 14:40:17.447798: train_loss -0.7575 
+2024-09-10 14:40:17.456855: val_loss -0.7581 
+2024-09-10 14:40:17.457907: Pseudo dice [0.9646, 0.9609, 0.9779, 0.9507, 0.8819, 0.9738, 0.9921] 
+2024-09-10 14:40:17.458663: Epoch time: 354.12 s 
+2024-09-10 14:40:21.229109:  
+2024-09-10 14:40:21.230293: Epoch 756 
+2024-09-10 14:40:21.230962: Current learning rate: 0.00281 
+2024-09-10 14:45:46.521513: train_loss -0.7403 
+2024-09-10 14:45:46.536389: val_loss -0.7472 
+2024-09-10 14:45:46.537924: Pseudo dice [0.9669, 0.9606, 0.9805, 0.9564, 0.8911, 0.9713, 0.9921] 
+2024-09-10 14:45:46.539620: Epoch time: 325.3 s 
+2024-09-10 14:45:49.950255:  
+2024-09-10 14:45:49.951690: Epoch 757 
+2024-09-10 14:45:49.952878: Current learning rate: 0.0028 
+2024-09-10 14:51:16.818525: train_loss -0.7511 
+2024-09-10 14:51:16.834203: val_loss -0.7641 
+2024-09-10 14:51:16.835614: Pseudo dice [0.9667, 0.9592, 0.982, 0.9516, 0.8839, 0.9749, 0.9912] 
+2024-09-10 14:51:16.836935: Epoch time: 326.87 s 
+2024-09-10 14:51:20.735552:  
+2024-09-10 14:51:20.752663: Epoch 758 
+2024-09-10 14:51:20.754129: Current learning rate: 0.00279 
+2024-09-10 14:56:24.395454: train_loss -0.7564 
+2024-09-10 14:56:24.416099: val_loss -0.7224 
+2024-09-10 14:56:24.417122: Pseudo dice [0.9679, 0.9595, 0.9794, 0.9508, 0.8783, 0.9643, 0.9899] 
+2024-09-10 14:56:24.418055: Epoch time: 303.67 s 
+2024-09-10 14:56:28.662873:  
+2024-09-10 14:56:28.684591: Epoch 759 
+2024-09-10 14:56:28.685510: Current learning rate: 0.00278 
+2024-09-10 15:01:52.406280: train_loss -0.7484 
+2024-09-10 15:01:52.409988: val_loss -0.7525 
+2024-09-10 15:01:52.411182: Pseudo dice [0.9654, 0.9603, 0.9795, 0.9563, 0.8917, 0.9746, 0.9916] 
+2024-09-10 15:01:52.412645: Epoch time: 323.75 s 
+2024-09-10 15:01:56.365463:  
+2024-09-10 15:01:56.366586: Epoch 760 
+2024-09-10 15:01:56.367866: Current learning rate: 0.00277 
+2024-09-10 15:07:18.073082: train_loss -0.7636 
+2024-09-10 15:07:18.081700: val_loss -0.7816 
+2024-09-10 15:07:18.082945: Pseudo dice [0.9691, 0.9635, 0.9834, 0.9566, 0.8838, 0.9748, 0.992] 
+2024-09-10 15:07:18.084403: Epoch time: 321.71 s 
+2024-09-10 15:07:21.983414:  
+2024-09-10 15:07:21.984714: Epoch 761 
+2024-09-10 15:07:21.985811: Current learning rate: 0.00276 
+2024-09-10 15:12:57.292038: train_loss -0.7433 
+2024-09-10 15:12:57.298297: val_loss -0.7455 
+2024-09-10 15:12:57.299958: Pseudo dice [0.9671, 0.9601, 0.9801, 0.9547, 0.8881, 0.9726, 0.9912] 
+2024-09-10 15:12:57.301045: Epoch time: 335.31 s 
+2024-09-10 15:12:57.302062: Yayy! New best EMA pseudo Dice: 0.958 
+2024-09-10 15:13:02.529963:  
+2024-09-10 15:13:02.531058: Epoch 762 
+2024-09-10 15:13:02.531790: Current learning rate: 0.00275 
+2024-09-10 15:18:05.854571: train_loss -0.7561 
+2024-09-10 15:18:05.857706: val_loss -0.7443 
+2024-09-10 15:18:05.858592: Pseudo dice [0.9684, 0.9626, 0.9812, 0.9534, 0.8809, 0.9724, 0.9905] 
+2024-09-10 15:18:05.859487: Epoch time: 303.33 s 
+2024-09-10 15:18:05.860255: Yayy! New best EMA pseudo Dice: 0.9581 
+2024-09-10 15:18:11.699374:  
+2024-09-10 15:18:11.700419: Epoch 763 
+2024-09-10 15:18:11.701123: Current learning rate: 0.00274 
+2024-09-10 15:23:19.471516: train_loss -0.7555 
+2024-09-10 15:23:19.484194: val_loss -0.7173 
+2024-09-10 15:23:19.485199: Pseudo dice [0.9632, 0.9587, 0.9726, 0.9493, 0.8791, 0.9679, 0.9894] 
+2024-09-10 15:23:19.486158: Epoch time: 307.78 s 
+2024-09-10 15:23:26.965884:  
+2024-09-10 15:23:26.966797: Epoch 764 
+2024-09-10 15:23:26.967412: Current learning rate: 0.00273 
+2024-09-10 15:28:27.349823: train_loss -0.7574 
+2024-09-10 15:28:27.358124: val_loss -0.769 
+2024-09-10 15:28:27.358959: Pseudo dice [0.9661, 0.9609, 0.9818, 0.9472, 0.8757, 0.9709, 0.99] 
+2024-09-10 15:28:27.359738: Epoch time: 300.39 s 
+2024-09-10 15:28:30.805175:  
+2024-09-10 15:28:30.816907: Epoch 765 
+2024-09-10 15:28:30.818031: Current learning rate: 0.00272 
+2024-09-10 15:33:49.443150: train_loss -0.7559 
+2024-09-10 15:33:49.462296: val_loss -0.7388 
+2024-09-10 15:33:49.463255: Pseudo dice [0.9661, 0.9576, 0.9734, 0.9544, 0.8843, 0.974, 0.9923] 
+2024-09-10 15:33:49.464047: Epoch time: 318.65 s 
+2024-09-10 15:33:53.000027:  
+2024-09-10 15:33:53.017675: Epoch 766 
+2024-09-10 15:33:53.019699: Current learning rate: 0.00271 
+2024-09-10 15:39:22.040284: train_loss -0.746 
+2024-09-10 15:39:22.057331: val_loss -0.721 
+2024-09-10 15:39:22.058513: Pseudo dice [0.9651, 0.9579, 0.9782, 0.9498, 0.874, 0.9712, 0.9918] 
+2024-09-10 15:39:22.059448: Epoch time: 329.07 s 
+2024-09-10 15:39:26.165140:  
+2024-09-10 15:39:26.166705: Epoch 767 
+2024-09-10 15:39:26.167719: Current learning rate: 0.0027 
+2024-09-10 15:44:44.581660: train_loss -0.7656 
+2024-09-10 15:44:44.584445: val_loss -0.7351 
+2024-09-10 15:44:44.586183: Pseudo dice [0.9665, 0.9591, 0.9809, 0.95, 0.8766, 0.9702, 0.9903] 
+2024-09-10 15:44:44.587681: Epoch time: 318.42 s 
+2024-09-10 15:44:49.682264:  
+2024-09-10 15:44:49.693428: Epoch 768 
+2024-09-10 15:44:49.700503: Current learning rate: 0.00268 
+2024-09-10 15:50:16.199108: train_loss -0.7538 
+2024-09-10 15:50:16.214719: val_loss -0.7194 
+2024-09-10 15:50:16.215862: Pseudo dice [0.9673, 0.9631, 0.98, 0.9532, 0.8877, 0.9693, 0.9914] 
+2024-09-10 15:50:16.216904: Epoch time: 326.53 s 
+2024-09-10 15:50:20.401755:  
+2024-09-10 15:50:20.403061: Epoch 769 
+2024-09-10 15:50:20.403856: Current learning rate: 0.00267 
+2024-09-10 15:55:31.000018: train_loss -0.7478 
+2024-09-10 15:55:31.025123: val_loss -0.7771 
+2024-09-10 15:55:31.025965: Pseudo dice [0.9659, 0.9603, 0.9778, 0.9536, 0.8852, 0.974, 0.9932] 
+2024-09-10 15:55:31.026724: Epoch time: 310.62 s 
+2024-09-10 15:55:35.089316:  
+2024-09-10 15:55:35.100553: Epoch 770 
+2024-09-10 15:55:35.101502: Current learning rate: 0.00266 
+2024-09-10 16:00:59.098639: train_loss -0.7632 
+2024-09-10 16:00:59.117647: val_loss -0.7368 
+2024-09-10 16:00:59.120845: Pseudo dice [0.9661, 0.9599, 0.9795, 0.9558, 0.8858, 0.9754, 0.9915] 
+2024-09-10 16:00:59.121758: Epoch time: 324.02 s 
+2024-09-10 16:01:02.128864:  
+2024-09-10 16:01:02.131096: Epoch 771 
+2024-09-10 16:01:02.133176: Current learning rate: 0.00265 
+2024-09-10 16:07:15.680054: train_loss -0.7497 
+2024-09-10 16:07:15.696036: val_loss -0.7688 
+2024-09-10 16:07:15.697366: Pseudo dice [0.9667, 0.9607, 0.9802, 0.9527, 0.8866, 0.973, 0.9919] 
+2024-09-10 16:07:15.698737: Epoch time: 373.57 s 
+2024-09-10 16:07:21.786573:  
+2024-09-10 16:07:21.800867: Epoch 772 
+2024-09-10 16:07:21.802254: Current learning rate: 0.00264 
+2024-09-10 16:13:12.975405: train_loss -0.7518 
+2024-09-10 16:13:12.979225: val_loss -0.7257 
+2024-09-10 16:13:12.980200: Pseudo dice [0.967, 0.9605, 0.9789, 0.9552, 0.8754, 0.9729, 0.9904] 
+2024-09-10 16:13:12.981109: Epoch time: 351.2 s 
+2024-09-10 16:13:16.684483:  
+2024-09-10 16:13:16.685725: Epoch 773 
+2024-09-10 16:13:16.686542: Current learning rate: 0.00263 
+2024-09-10 16:19:34.404686: train_loss -0.7638 
+2024-09-10 16:19:34.423745: val_loss -0.7528 
+2024-09-10 16:19:34.425227: Pseudo dice [0.9675, 0.9599, 0.9809, 0.946, 0.8869, 0.9757, 0.9926] 
+2024-09-10 16:19:34.426459: Epoch time: 377.73 s 
+2024-09-10 16:19:39.533136:  
+2024-09-10 16:19:39.550072: Epoch 774 
+2024-09-10 16:19:39.553699: Current learning rate: 0.00262 
+2024-09-10 16:25:27.145916: train_loss -0.7525 
+2024-09-10 16:25:27.147061: val_loss -0.7358 
+2024-09-10 16:25:27.147926: Pseudo dice [0.9658, 0.9596, 0.9803, 0.9542, 0.8943, 0.9751, 0.9917] 
+2024-09-10 16:25:27.148573: Epoch time: 347.68 s 
+2024-09-10 16:25:32.311577:  
+2024-09-10 16:25:32.324703: Epoch 775 
+2024-09-10 16:25:32.325664: Current learning rate: 0.00261 
+2024-09-10 16:31:47.686040: train_loss -0.749 
+2024-09-10 16:31:47.714433: val_loss -0.7339 
+2024-09-10 16:31:47.715873: Pseudo dice [0.9649, 0.9607, 0.9774, 0.9526, 0.8801, 0.9715, 0.9914] 
+2024-09-10 16:31:47.717080: Epoch time: 375.38 s 
+2024-09-10 16:31:51.068707:  
+2024-09-10 16:31:51.070547: Epoch 776 
+2024-09-10 16:31:51.073613: Current learning rate: 0.0026 
+2024-09-10 16:37:36.753850: train_loss -0.7733 
+2024-09-10 16:37:36.769849: val_loss -0.7423 
+2024-09-10 16:37:36.770799: Pseudo dice [0.9656, 0.9601, 0.9771, 0.9509, 0.8816, 0.9707, 0.991] 
+2024-09-10 16:37:36.771560: Epoch time: 345.72 s 
+2024-09-10 16:37:39.762406:  
+2024-09-10 16:37:39.763832: Epoch 777 
+2024-09-10 16:37:39.765234: Current learning rate: 0.00259 
+2024-09-10 16:43:26.462552: train_loss -0.747 
+2024-09-10 16:43:26.481086: val_loss -0.7805 
+2024-09-10 16:43:26.482179: Pseudo dice [0.9657, 0.9609, 0.9792, 0.9472, 0.8825, 0.9734, 0.9919] 
+2024-09-10 16:43:26.483065: Epoch time: 346.71 s 
+2024-09-10 16:43:34.113965:  
+2024-09-10 16:43:34.115405: Epoch 778 
+2024-09-10 16:43:34.116257: Current learning rate: 0.00258 
+2024-09-10 16:49:37.445837: train_loss -0.7484 
+2024-09-10 16:49:37.464026: val_loss -0.768 
+2024-09-10 16:49:37.465849: Pseudo dice [0.9654, 0.9579, 0.9801, 0.9524, 0.8771, 0.9723, 0.9908] 
+2024-09-10 16:49:37.467586: Epoch time: 363.34 s 
+2024-09-10 16:49:41.264227:  
+2024-09-10 16:49:41.281134: Epoch 779 
+2024-09-10 16:49:41.282037: Current learning rate: 0.00257 
+2024-09-10 16:55:11.935289: train_loss -0.7504 
+2024-09-10 16:55:11.949516: val_loss -0.7432 
+2024-09-10 16:55:11.950313: Pseudo dice [0.9652, 0.9593, 0.9792, 0.9477, 0.8691, 0.9646, 0.9891] 
+2024-09-10 16:55:11.951024: Epoch time: 330.68 s 
+2024-09-10 16:55:15.440273:  
+2024-09-10 16:55:15.452744: Epoch 780 
+2024-09-10 16:55:15.454142: Current learning rate: 0.00256 
+2024-09-10 17:00:43.984660: train_loss -0.7425 
+2024-09-10 17:00:43.987951: val_loss -0.7257 
+2024-09-10 17:00:43.988902: Pseudo dice [0.9642, 0.9562, 0.9792, 0.9488, 0.8757, 0.9665, 0.9898] 
+2024-09-10 17:00:43.989901: Epoch time: 328.55 s 
+2024-09-10 17:00:48.141197:  
+2024-09-10 17:00:48.155814: Epoch 781 
+2024-09-10 17:00:48.157258: Current learning rate: 0.00255 
+2024-09-10 17:06:33.287584: train_loss -0.7595 
+2024-09-10 17:06:33.290191: val_loss -0.765 
+2024-09-10 17:06:33.291116: Pseudo dice [0.9654, 0.9603, 0.9791, 0.9535, 0.8815, 0.9729, 0.9898] 
+2024-09-10 17:06:33.292076: Epoch time: 345.15 s 
+2024-09-10 17:06:37.961965:  
+2024-09-10 17:06:37.963621: Epoch 782 
+2024-09-10 17:06:37.965865: Current learning rate: 0.00254 
+2024-09-10 17:12:34.668179: train_loss -0.7454 
+2024-09-10 17:12:34.670557: val_loss -0.7373 
+2024-09-10 17:12:34.671575: Pseudo dice [0.9675, 0.9608, 0.9807, 0.9488, 0.8822, 0.9725, 0.9898] 
+2024-09-10 17:12:34.672390: Epoch time: 356.71 s 
+2024-09-10 17:12:37.200679:  
+2024-09-10 17:12:37.201802: Epoch 783 
+2024-09-10 17:12:37.202647: Current learning rate: 0.00253 
+2024-09-10 17:18:13.629493: train_loss -0.7491 
+2024-09-10 17:18:13.631454: val_loss -0.7508 
+2024-09-10 17:18:13.632145: Pseudo dice [0.9684, 0.9606, 0.9809, 0.9483, 0.8873, 0.9721, 0.993] 
+2024-09-10 17:18:13.632806: Epoch time: 336.43 s 
+2024-09-10 17:18:16.668499:  
+2024-09-10 17:18:16.682928: Epoch 784 
+2024-09-10 17:18:16.685323: Current learning rate: 0.00252 
+2024-09-10 17:24:01.678583: train_loss -0.7477 
+2024-09-10 17:24:01.714487: val_loss -0.7559 
+2024-09-10 17:24:01.715734: Pseudo dice [0.9655, 0.9606, 0.9793, 0.956, 0.889, 0.9746, 0.9913] 
+2024-09-10 17:24:01.716669: Epoch time: 345.02 s 
+2024-09-10 17:24:05.578907:  
+2024-09-10 17:24:05.595757: Epoch 785 
+2024-09-10 17:24:05.596767: Current learning rate: 0.00251 
+2024-09-10 17:30:13.550675: train_loss -0.7442 
+2024-09-10 17:30:13.566168: val_loss -0.7344 
+2024-09-10 17:30:13.567394: Pseudo dice [0.9686, 0.9606, 0.9837, 0.9516, 0.8909, 0.9738, 0.9921] 
+2024-09-10 17:30:13.568320: Epoch time: 367.98 s 
+2024-09-10 17:30:16.495269:  
+2024-09-10 17:30:16.496487: Epoch 786 
+2024-09-10 17:30:16.497365: Current learning rate: 0.0025 
+2024-09-10 17:36:27.017260: train_loss -0.7464 
+2024-09-10 17:36:27.033589: val_loss -0.7507 
+2024-09-10 17:36:27.034370: Pseudo dice [0.9682, 0.9597, 0.9824, 0.9464, 0.8901, 0.9719, 0.9915] 
+2024-09-10 17:36:27.035021: Epoch time: 370.55 s 
+2024-09-10 17:36:32.425173:  
+2024-09-10 17:36:32.426262: Epoch 787 
+2024-09-10 17:36:32.426908: Current learning rate: 0.00249 
+2024-09-10 17:42:57.237439: train_loss -0.7412 
+2024-09-10 17:42:57.247410: val_loss -0.7216 
+2024-09-10 17:42:57.248407: Pseudo dice [0.9681, 0.9609, 0.9805, 0.9536, 0.8814, 0.9726, 0.9919] 
+2024-09-10 17:42:57.249103: Epoch time: 384.82 s 
+2024-09-10 17:43:01.741999:  
+2024-09-10 17:43:01.759870: Epoch 788 
+2024-09-10 17:43:01.761583: Current learning rate: 0.00248 
+2024-09-10 17:48:45.775778: train_loss -0.7669 
+2024-09-10 17:48:45.792547: val_loss -0.7731 
+2024-09-10 17:48:45.794382: Pseudo dice [0.9654, 0.9613, 0.9764, 0.9555, 0.8889, 0.9734, 0.9927] 
+2024-09-10 17:48:45.795994: Epoch time: 344.04 s 
+2024-09-10 17:48:50.527246:  
+2024-09-10 17:48:50.529085: Epoch 789 
+2024-09-10 17:48:50.529991: Current learning rate: 0.00247 
+2024-09-10 17:54:57.031866: train_loss -0.7536 
+2024-09-10 17:54:57.043720: val_loss -0.7333 
+2024-09-10 17:54:57.045006: Pseudo dice [0.9661, 0.9596, 0.9811, 0.9564, 0.886, 0.9752, 0.9916] 
+2024-09-10 17:54:57.046297: Epoch time: 366.51 s 
+2024-09-10 17:54:57.047377: Yayy! New best EMA pseudo Dice: 0.9581 
+2024-09-10 17:55:05.024088:  
+2024-09-10 17:55:05.026838: Epoch 790 
+2024-09-10 17:55:05.027666: Current learning rate: 0.00245 
+2024-09-10 18:01:08.045951: train_loss -0.741 
+2024-09-10 18:01:08.066731: val_loss -0.7362 
+2024-09-10 18:01:08.067589: Pseudo dice [0.9636, 0.9613, 0.9762, 0.9504, 0.8898, 0.9711, 0.9912] 
+2024-09-10 18:01:08.068248: Epoch time: 363.03 s 
+2024-09-10 18:01:13.553292:  
+2024-09-10 18:01:13.554456: Epoch 791 
+2024-09-10 18:01:13.555285: Current learning rate: 0.00244 
+2024-09-10 18:06:54.397203: train_loss -0.756 
+2024-09-10 18:06:54.412700: val_loss -0.7716 
+2024-09-10 18:06:54.413638: Pseudo dice [0.9653, 0.9609, 0.9778, 0.9513, 0.891, 0.9738, 0.9929] 
+2024-09-10 18:06:54.414823: Epoch time: 340.85 s 
+2024-09-10 18:06:54.415619: Yayy! New best EMA pseudo Dice: 0.9582 
+2024-09-10 18:07:02.630214:  
+2024-09-10 18:07:02.631558: Epoch 792 
+2024-09-10 18:07:02.632483: Current learning rate: 0.00243 
+2024-09-10 18:12:44.424041: train_loss -0.7529 
+2024-09-10 18:12:44.437725: val_loss -0.7191 
+2024-09-10 18:12:44.438604: Pseudo dice [0.9654, 0.958, 0.9807, 0.9525, 0.8888, 0.9736, 0.9914] 
+2024-09-10 18:12:44.439799: Epoch time: 341.8 s 
+2024-09-10 18:12:44.440570: Yayy! New best EMA pseudo Dice: 0.9582 
+2024-09-10 18:12:50.610886:  
+2024-09-10 18:12:50.612005: Epoch 793 
+2024-09-10 18:12:50.612802: Current learning rate: 0.00242 
+2024-09-10 18:18:38.073766: train_loss -0.7653 
+2024-09-10 18:18:38.075972: val_loss -0.7689 
+2024-09-10 18:18:38.076772: Pseudo dice [0.9641, 0.9616, 0.9779, 0.9546, 0.893, 0.9729, 0.9911] 
+2024-09-10 18:18:38.077584: Epoch time: 347.47 s 
+2024-09-10 18:18:38.078228: Yayy! New best EMA pseudo Dice: 0.9583 
+2024-09-10 18:18:45.529763:  
+2024-09-10 18:18:45.531196: Epoch 794 
+2024-09-10 18:18:45.532077: Current learning rate: 0.00241 
+2024-09-10 18:24:46.789898: train_loss -0.7541 
+2024-09-10 18:24:46.796895: val_loss -0.775 
+2024-09-10 18:24:46.798059: Pseudo dice [0.968, 0.961, 0.9819, 0.956, 0.8918, 0.9734, 0.9918] 
+2024-09-10 18:24:46.799569: Epoch time: 361.27 s 
+2024-09-10 18:24:46.800295: Yayy! New best EMA pseudo Dice: 0.9585 
+2024-09-10 18:24:55.149126:  
+2024-09-10 18:24:55.150212: Epoch 795 
+2024-09-10 18:24:55.151265: Current learning rate: 0.0024 
+2024-09-10 18:30:43.256553: train_loss -0.7525 
+2024-09-10 18:30:43.258993: val_loss -0.7736 
+2024-09-10 18:30:43.259969: Pseudo dice [0.9692, 0.9606, 0.9811, 0.9558, 0.899, 0.9762, 0.9929] 
+2024-09-10 18:30:43.261051: Epoch time: 348.11 s 
+2024-09-10 18:30:43.261737: Yayy! New best EMA pseudo Dice: 0.9589 
+2024-09-10 18:30:48.945561:  
+2024-09-10 18:30:48.946953: Epoch 796 
+2024-09-10 18:30:48.947547: Current learning rate: 0.00239 
+2024-09-10 18:36:46.325408: train_loss -0.7625 
+2024-09-10 18:36:46.331793: val_loss -0.7743 
+2024-09-10 18:36:46.333856: Pseudo dice [0.9692, 0.9613, 0.9827, 0.952, 0.8863, 0.9747, 0.992] 
+2024-09-10 18:36:46.335740: Epoch time: 357.38 s 
+2024-09-10 18:36:46.337752: Yayy! New best EMA pseudo Dice: 0.959 
+2024-09-10 18:36:55.598679:  
+2024-09-10 18:36:55.599812: Epoch 797 
+2024-09-10 18:36:55.601157: Current learning rate: 0.00238 
+2024-09-10 18:42:35.597479: train_loss -0.754 
+2024-09-10 18:42:35.612140: val_loss -0.7508 
+2024-09-10 18:42:35.613616: Pseudo dice [0.9696, 0.9638, 0.9817, 0.9588, 0.8827, 0.9711, 0.9934] 
+2024-09-10 18:42:35.614598: Epoch time: 340.0 s 
+2024-09-10 18:42:35.615371: Yayy! New best EMA pseudo Dice: 0.9591 
+2024-09-10 18:42:42.208056:  
+2024-09-10 18:42:42.209141: Epoch 798 
+2024-09-10 18:42:42.209815: Current learning rate: 0.00237 
+2024-09-10 18:48:32.330175: train_loss -0.7501 
+2024-09-10 18:48:32.341763: val_loss -0.7334 
+2024-09-10 18:48:32.342779: Pseudo dice [0.9654, 0.9614, 0.9765, 0.9503, 0.8835, 0.9741, 0.99] 
+2024-09-10 18:48:32.343461: Epoch time: 350.13 s 
+2024-09-10 18:48:35.693275:  
+2024-09-10 18:48:35.708390: Epoch 799 
+2024-09-10 18:48:35.709264: Current learning rate: 0.00236 
+2024-09-10 18:54:19.490940: train_loss -0.7615 
+2024-09-10 18:54:19.516934: val_loss -0.7423 
+2024-09-10 18:54:19.520457: Pseudo dice [0.9616, 0.9571, 0.9757, 0.948, 0.8826, 0.972, 0.9922] 
+2024-09-10 18:54:19.521461: Epoch time: 343.8 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_10_22_43_26.txt

@@ -0,0 +1,735 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-10 22:43:37.507574: Using torch.compile... 
+2024-09-10 22:43:49.296293: do_dummy_2d_data_aug: False 
+2024-09-10 22:43:49.302640: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-10 22:43:49.304840: The split file contains 5 splits. 
+2024-09-10 22:43:49.305415: Desired fold for training: 0 
+2024-09-10 22:43:49.305972: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-10 22:44:01.158609: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-10 22:44:05.169764:  
+2024-09-10 22:44:05.171315: Epoch 800 
+2024-09-10 22:44:05.172596: Current learning rate: 0.00235 
+2024-09-10 22:57:36.856871: train_loss -0.7647 
+2024-09-10 22:57:36.858979: val_loss -0.7372 
+2024-09-10 22:57:36.859646: Pseudo dice [0.9677, 0.9605, 0.9831, 0.9511, 0.8915, 0.9743, 0.9921] 
+2024-09-10 22:57:36.860265: Epoch time: 811.69 s 
+2024-09-10 22:57:38.347289:  
+2024-09-10 22:57:38.348130: Epoch 801 
+2024-09-10 22:57:38.348826: Current learning rate: 0.00234 
+2024-09-10 23:02:18.484663: train_loss -0.7571 
+2024-09-10 23:02:18.497958: val_loss -0.7453 
+2024-09-10 23:02:18.498791: Pseudo dice [0.9689, 0.9612, 0.9823, 0.9571, 0.8917, 0.9745, 0.9905] 
+2024-09-10 23:02:18.499503: Epoch time: 280.14 s 
+2024-09-10 23:02:21.026017:  
+2024-09-10 23:02:21.026958: Epoch 802 
+2024-09-10 23:02:21.027874: Current learning rate: 0.00233 
+2024-09-10 23:06:52.324094: train_loss -0.7584 
+2024-09-10 23:06:52.345623: val_loss -0.786 
+2024-09-10 23:06:52.346612: Pseudo dice [0.9675, 0.9619, 0.9807, 0.9522, 0.884, 0.9715, 0.9885] 
+2024-09-10 23:06:52.347515: Epoch time: 271.3 s 
+2024-09-10 23:06:54.858696:  
+2024-09-10 23:06:54.860654: Epoch 803 
+2024-09-10 23:06:54.861854: Current learning rate: 0.00232 
+2024-09-10 23:11:17.185200: train_loss -0.7499 
+2024-09-10 23:11:17.198082: val_loss -0.7381 
+2024-09-10 23:11:17.198870: Pseudo dice [0.9643, 0.9602, 0.9768, 0.9556, 0.877, 0.9711, 0.9895] 
+2024-09-10 23:11:17.199747: Epoch time: 262.33 s 
+2024-09-10 23:11:20.789693:  
+2024-09-10 23:11:20.803706: Epoch 804 
+2024-09-10 23:11:20.804586: Current learning rate: 0.00231 
+2024-09-10 23:15:48.485649: train_loss -0.7512 
+2024-09-10 23:15:48.496685: val_loss -0.7636 
+2024-09-10 23:15:48.497454: Pseudo dice [0.9684, 0.9593, 0.9821, 0.9554, 0.8923, 0.9741, 0.9917] 
+2024-09-10 23:15:48.498261: Epoch time: 267.7 s 
+2024-09-10 23:15:51.749179:  
+2024-09-10 23:15:51.764097: Epoch 805 
+2024-09-10 23:15:51.765172: Current learning rate: 0.0023 
+2024-09-10 23:20:16.105614: train_loss -0.7525 
+2024-09-10 23:20:16.107822: val_loss -0.7547 
+2024-09-10 23:20:16.108618: Pseudo dice [0.9652, 0.9621, 0.9789, 0.9473, 0.8768, 0.9716, 0.9897] 
+2024-09-10 23:20:16.109310: Epoch time: 264.36 s 
+2024-09-10 23:20:19.159151:  
+2024-09-10 23:20:19.160056: Epoch 806 
+2024-09-10 23:20:19.160824: Current learning rate: 0.00229 
+2024-09-10 23:24:51.429153: train_loss -0.7748 
+2024-09-10 23:24:51.446084: val_loss -0.7709 
+2024-09-10 23:24:51.446886: Pseudo dice [0.9663, 0.9624, 0.9797, 0.9568, 0.8849, 0.9761, 0.9907] 
+2024-09-10 23:24:51.447602: Epoch time: 272.27 s 
+2024-09-10 23:24:55.168958:  
+2024-09-10 23:24:55.184758: Epoch 807 
+2024-09-10 23:24:55.185578: Current learning rate: 0.00228 
+2024-09-10 23:29:23.051042: train_loss -0.7623 
+2024-09-10 23:29:23.065700: val_loss -0.7338 
+2024-09-10 23:29:23.066477: Pseudo dice [0.9692, 0.9631, 0.9826, 0.9511, 0.8928, 0.9753, 0.9928] 
+2024-09-10 23:29:23.067184: Epoch time: 267.89 s 
+2024-09-10 23:29:25.706478:  
+2024-09-10 23:29:25.707392: Epoch 808 
+2024-09-10 23:29:25.708046: Current learning rate: 0.00226 
+2024-09-10 23:33:50.048418: train_loss -0.7487 
+2024-09-10 23:33:50.066668: val_loss -0.7713 
+2024-09-10 23:33:50.067729: Pseudo dice [0.9665, 0.9593, 0.9807, 0.9558, 0.8907, 0.9754, 0.9917] 
+2024-09-10 23:33:50.068772: Epoch time: 264.35 s 
+2024-09-10 23:33:54.328150:  
+2024-09-10 23:33:54.329012: Epoch 809 
+2024-09-10 23:33:54.329691: Current learning rate: 0.00225 
+2024-09-10 23:38:23.070516: train_loss -0.7411 
+2024-09-10 23:38:23.084304: val_loss -0.7845 
+2024-09-10 23:38:23.085688: Pseudo dice [0.9686, 0.9614, 0.9823, 0.9547, 0.896, 0.9756, 0.9928] 
+2024-09-10 23:38:23.086539: Epoch time: 268.75 s 
+2024-09-10 23:38:23.087473: Yayy! New best EMA pseudo Dice: 0.9592 
+2024-09-10 23:38:29.791807:  
+2024-09-10 23:38:29.792677: Epoch 810 
+2024-09-10 23:38:29.793334: Current learning rate: 0.00224 
+2024-09-10 23:42:48.615979: train_loss -0.7364 
+2024-09-10 23:42:48.634731: val_loss -0.7641 
+2024-09-10 23:42:48.635605: Pseudo dice [0.9648, 0.9585, 0.9805, 0.9541, 0.887, 0.973, 0.9905] 
+2024-09-10 23:42:48.636373: Epoch time: 258.83 s 
+2024-09-10 23:42:51.569154:  
+2024-09-10 23:42:51.570195: Epoch 811 
+2024-09-10 23:42:51.570883: Current learning rate: 0.00223 
+2024-09-10 23:47:23.033719: train_loss -0.7397 
+2024-09-10 23:47:23.049904: val_loss -0.7751 
+2024-09-10 23:47:23.050691: Pseudo dice [0.966, 0.9601, 0.977, 0.9522, 0.893, 0.9724, 0.9917] 
+2024-09-10 23:47:23.051404: Epoch time: 271.47 s 
+2024-09-10 23:47:25.972187:  
+2024-09-10 23:47:25.973094: Epoch 812 
+2024-09-10 23:47:25.973817: Current learning rate: 0.00222 
+2024-09-10 23:51:29.205055: train_loss -0.7377 
+2024-09-10 23:51:29.222021: val_loss -0.793 
+2024-09-10 23:51:29.222969: Pseudo dice [0.9685, 0.9625, 0.9825, 0.9507, 0.8886, 0.9738, 0.9915] 
+2024-09-10 23:51:29.223695: Epoch time: 243.24 s 
+2024-09-10 23:51:34.440753:  
+2024-09-10 23:51:34.441726: Epoch 813 
+2024-09-10 23:51:34.442343: Current learning rate: 0.00221 
+2024-09-10 23:56:01.402431: train_loss -0.7517 
+2024-09-10 23:56:01.414263: val_loss -0.7667 
+2024-09-10 23:56:01.415147: Pseudo dice [0.9641, 0.9602, 0.9785, 0.9539, 0.8827, 0.9689, 0.9896] 
+2024-09-10 23:56:01.415926: Epoch time: 266.97 s 
+2024-09-10 23:56:04.589951:  
+2024-09-10 23:56:04.590808: Epoch 814 
+2024-09-10 23:56:04.591452: Current learning rate: 0.0022 
+2024-09-11 00:00:24.989139: train_loss -0.7404 
+2024-09-11 00:00:24.991754: val_loss -0.7559 
+2024-09-11 00:00:25.009501: Pseudo dice [0.9664, 0.9595, 0.9807, 0.955, 0.8926, 0.9691, 0.9899] 
+2024-09-11 00:00:25.011135: Epoch time: 260.4 s 
+2024-09-11 00:00:28.500921:  
+2024-09-11 00:00:28.502079: Epoch 815 
+2024-09-11 00:00:28.502777: Current learning rate: 0.00219 
+2024-09-11 00:05:00.014333: train_loss -0.7489 
+2024-09-11 00:05:00.026152: val_loss -0.7143 
+2024-09-11 00:05:00.027280: Pseudo dice [0.9671, 0.9593, 0.9809, 0.9533, 0.8906, 0.9702, 0.9889] 
+2024-09-11 00:05:00.028013: Epoch time: 271.52 s 
+2024-09-11 00:05:03.694023:  
+2024-09-11 00:05:03.695045: Epoch 816 
+2024-09-11 00:05:03.695799: Current learning rate: 0.00218 
+2024-09-11 00:09:19.007727: train_loss -0.7662 
+2024-09-11 00:09:19.028161: val_loss -0.7792 
+2024-09-11 00:09:19.029060: Pseudo dice [0.9686, 0.9622, 0.9824, 0.9588, 0.878, 0.9747, 0.9908] 
+2024-09-11 00:09:19.029722: Epoch time: 255.32 s 
+2024-09-11 00:09:21.884318:  
+2024-09-11 00:09:21.885164: Epoch 817 
+2024-09-11 00:09:21.885860: Current learning rate: 0.00217 
+2024-09-11 00:13:34.059092: train_loss -0.7588 
+2024-09-11 00:13:34.061319: val_loss -0.7421 
+2024-09-11 00:13:34.062016: Pseudo dice [0.9686, 0.9611, 0.9808, 0.9494, 0.8919, 0.9749, 0.9926] 
+2024-09-11 00:13:34.062727: Epoch time: 252.18 s 
+2024-09-11 00:13:36.340027:  
+2024-09-11 00:13:36.340977: Epoch 818 
+2024-09-11 00:13:36.341695: Current learning rate: 0.00216 
+2024-09-11 00:18:09.100689: train_loss -0.7569 
+2024-09-11 00:18:09.102839: val_loss -0.7736 
+2024-09-11 00:18:09.103541: Pseudo dice [0.967, 0.9611, 0.9794, 0.954, 0.8875, 0.974, 0.9893] 
+2024-09-11 00:18:09.116219: Epoch time: 272.77 s 
+2024-09-11 00:18:12.361102:  
+2024-09-11 00:18:12.372544: Epoch 819 
+2024-09-11 00:18:12.373238: Current learning rate: 0.00215 
+2024-09-11 00:22:39.563144: train_loss -0.7571 
+2024-09-11 00:22:39.577379: val_loss -0.7185 
+2024-09-11 00:22:39.578172: Pseudo dice [0.9673, 0.9606, 0.981, 0.945, 0.8868, 0.9724, 0.9913] 
+2024-09-11 00:22:39.578762: Epoch time: 267.21 s 
+2024-09-11 00:22:43.401114:  
+2024-09-11 00:22:43.402091: Epoch 820 
+2024-09-11 00:22:43.402835: Current learning rate: 0.00214 
+2024-09-11 00:27:08.992467: train_loss -0.7612 
+2024-09-11 00:27:08.994691: val_loss -0.7641 
+2024-09-11 00:27:08.995373: Pseudo dice [0.9691, 0.9614, 0.9811, 0.9554, 0.8916, 0.976, 0.9923] 
+2024-09-11 00:27:08.996118: Epoch time: 265.6 s 
+2024-09-11 00:27:12.413179:  
+2024-09-11 00:27:12.432688: Epoch 821 
+2024-09-11 00:27:12.433432: Current learning rate: 0.00213 
+2024-09-11 00:31:20.770205: train_loss -0.7537 
+2024-09-11 00:31:20.788205: val_loss -0.7491 
+2024-09-11 00:31:20.789159: Pseudo dice [0.964, 0.9569, 0.9785, 0.9499, 0.8805, 0.9755, 0.9926] 
+2024-09-11 00:31:20.789876: Epoch time: 248.36 s 
+2024-09-11 00:31:23.909444:  
+2024-09-11 00:31:23.910511: Epoch 822 
+2024-09-11 00:31:23.911250: Current learning rate: 0.00212 
+2024-09-11 00:35:52.163964: train_loss -0.7548 
+2024-09-11 00:35:52.182156: val_loss -0.74 
+2024-09-11 00:35:52.183042: Pseudo dice [0.9688, 0.9628, 0.9817, 0.9579, 0.8826, 0.9741, 0.9912] 
+2024-09-11 00:35:52.183705: Epoch time: 268.26 s 
+2024-09-11 00:35:54.965486:  
+2024-09-11 00:35:54.966423: Epoch 823 
+2024-09-11 00:35:54.967094: Current learning rate: 0.0021 
+2024-09-11 00:40:22.059033: train_loss -0.7633 
+2024-09-11 00:40:22.062001: val_loss -0.7467 
+2024-09-11 00:40:22.063321: Pseudo dice [0.9657, 0.9592, 0.9806, 0.9537, 0.8909, 0.9754, 0.9917] 
+2024-09-11 00:40:22.064241: Epoch time: 267.1 s 
+2024-09-11 00:40:25.591956:  
+2024-09-11 00:40:25.608759: Epoch 824 
+2024-09-11 00:40:25.609652: Current learning rate: 0.00209 
+2024-09-11 00:44:56.670736: train_loss -0.7703 
+2024-09-11 00:44:56.686031: val_loss -0.7715 
+2024-09-11 00:44:56.686783: Pseudo dice [0.9667, 0.9606, 0.9805, 0.9516, 0.8898, 0.9763, 0.9917] 
+2024-09-11 00:44:56.687420: Epoch time: 271.09 s 
+2024-09-11 00:45:00.105107:  
+2024-09-11 00:45:00.106144: Epoch 825 
+2024-09-11 00:45:00.106802: Current learning rate: 0.00208 
+2024-09-11 00:49:22.684099: train_loss -0.7659 
+2024-09-11 00:49:22.694318: val_loss -0.7498 
+2024-09-11 00:49:22.695263: Pseudo dice [0.9673, 0.9611, 0.982, 0.9578, 0.8925, 0.9747, 0.9914] 
+2024-09-11 00:49:22.695985: Epoch time: 262.58 s 
+2024-09-11 00:49:22.696851: Yayy! New best EMA pseudo Dice: 0.9593 
+2024-09-11 00:49:31.722444:  
+2024-09-11 00:49:31.723362: Epoch 826 
+2024-09-11 00:49:31.724037: Current learning rate: 0.00207 
+2024-09-11 00:54:01.507206: train_loss -0.7479 
+2024-09-11 00:54:01.521574: val_loss -0.7805 
+2024-09-11 00:54:01.522352: Pseudo dice [0.965, 0.9595, 0.9791, 0.9499, 0.8922, 0.9744, 0.9916] 
+2024-09-11 00:54:01.523014: Epoch time: 269.79 s 
+2024-09-11 00:54:04.135038:  
+2024-09-11 00:54:04.148698: Epoch 827 
+2024-09-11 00:54:04.149524: Current learning rate: 0.00206 
+2024-09-11 00:58:37.098170: train_loss -0.7504 
+2024-09-11 00:58:37.112507: val_loss -0.725 
+2024-09-11 00:58:37.113240: Pseudo dice [0.9681, 0.9603, 0.9806, 0.9578, 0.8902, 0.9749, 0.9924] 
+2024-09-11 00:58:37.113916: Epoch time: 272.97 s 
+2024-09-11 00:58:37.114523: Yayy! New best EMA pseudo Dice: 0.9594 
+2024-09-11 00:58:44.746222:  
+2024-09-11 00:58:44.747029: Epoch 828 
+2024-09-11 00:58:44.747711: Current learning rate: 0.00205 
+2024-09-11 01:03:19.491414: train_loss -0.7601 
+2024-09-11 01:03:19.494245: val_loss -0.7483 
+2024-09-11 01:03:19.495023: Pseudo dice [0.9676, 0.9609, 0.9802, 0.952, 0.8863, 0.9736, 0.9921] 
+2024-09-11 01:03:19.495642: Epoch time: 274.75 s 
+2024-09-11 01:03:22.536151:  
+2024-09-11 01:03:22.545611: Epoch 829 
+2024-09-11 01:03:22.546332: Current learning rate: 0.00204 
+2024-09-11 01:07:49.393862: train_loss -0.7655 
+2024-09-11 01:07:49.403459: val_loss -0.7827 
+2024-09-11 01:07:49.404456: Pseudo dice [0.9675, 0.958, 0.9804, 0.9555, 0.8981, 0.9773, 0.9927] 
+2024-09-11 01:07:49.405195: Epoch time: 266.86 s 
+2024-09-11 01:07:49.405799: Yayy! New best EMA pseudo Dice: 0.9595 
+2024-09-11 01:07:55.091319:  
+2024-09-11 01:07:55.092148: Epoch 830 
+2024-09-11 01:07:55.092742: Current learning rate: 0.00203 
+2024-09-11 01:12:21.152513: train_loss -0.7547 
+2024-09-11 01:12:21.154998: val_loss -0.7678 
+2024-09-11 01:12:21.156138: Pseudo dice [0.9654, 0.959, 0.9795, 0.9517, 0.8933, 0.9752, 0.9918] 
+2024-09-11 01:12:21.157058: Epoch time: 266.06 s 
+2024-09-11 01:12:23.545516:  
+2024-09-11 01:12:23.546422: Epoch 831 
+2024-09-11 01:12:23.547116: Current learning rate: 0.00202 
+2024-09-11 01:16:50.709289: train_loss -0.757 
+2024-09-11 01:16:50.711642: val_loss -0.7569 
+2024-09-11 01:16:50.712494: Pseudo dice [0.9684, 0.9592, 0.9828, 0.9533, 0.8953, 0.9756, 0.9921] 
+2024-09-11 01:16:50.713378: Epoch time: 267.17 s 
+2024-09-11 01:16:50.714199: Yayy! New best EMA pseudo Dice: 0.9597 
+2024-09-11 01:16:56.348132:  
+2024-09-11 01:16:56.348956: Epoch 832 
+2024-09-11 01:16:56.349635: Current learning rate: 0.00201 
+2024-09-11 01:21:35.984761: train_loss -0.7425 
+2024-09-11 01:21:36.002298: val_loss -0.7533 
+2024-09-11 01:21:36.003208: Pseudo dice [0.9678, 0.9608, 0.9825, 0.9543, 0.882, 0.9749, 0.9903] 
+2024-09-11 01:21:36.003866: Epoch time: 279.64 s 
+2024-09-11 01:21:39.113856:  
+2024-09-11 01:21:39.120610: Epoch 833 
+2024-09-11 01:21:39.121630: Current learning rate: 0.002 
+2024-09-11 01:26:09.698287: train_loss -0.7477 
+2024-09-11 01:26:09.710795: val_loss -0.7405 
+2024-09-11 01:26:09.711775: Pseudo dice [0.9678, 0.96, 0.9811, 0.9553, 0.8835, 0.9725, 0.9898] 
+2024-09-11 01:26:09.712612: Epoch time: 270.59 s 
+2024-09-11 01:26:11.886251:  
+2024-09-11 01:26:11.887302: Epoch 834 
+2024-09-11 01:26:11.888103: Current learning rate: 0.00199 
+2024-09-11 01:30:39.440245: train_loss -0.7615 
+2024-09-11 01:30:39.460171: val_loss -0.763 
+2024-09-11 01:30:39.461125: Pseudo dice [0.9663, 0.9605, 0.9796, 0.9517, 0.8899, 0.9743, 0.9918] 
+2024-09-11 01:30:39.461860: Epoch time: 267.56 s 
+2024-09-11 01:30:42.830239:  
+2024-09-11 01:30:42.831411: Epoch 835 
+2024-09-11 01:30:42.832217: Current learning rate: 0.00198 
+2024-09-11 01:35:03.724127: train_loss -0.7677 
+2024-09-11 01:35:03.742862: val_loss -0.7546 
+2024-09-11 01:35:03.743965: Pseudo dice [0.9655, 0.96, 0.9777, 0.9581, 0.8906, 0.9755, 0.9926] 
+2024-09-11 01:35:03.745112: Epoch time: 260.9 s 
+2024-09-11 01:35:07.141934:  
+2024-09-11 01:35:07.142874: Epoch 836 
+2024-09-11 01:35:07.143518: Current learning rate: 0.00196 
+2024-09-11 01:39:37.593487: train_loss -0.7638 
+2024-09-11 01:39:37.595499: val_loss -0.7827 
+2024-09-11 01:39:37.596048: Pseudo dice [0.9675, 0.9599, 0.9836, 0.9533, 0.894, 0.9735, 0.9908] 
+2024-09-11 01:39:37.596686: Epoch time: 270.46 s 
+2024-09-11 01:39:41.124541:  
+2024-09-11 01:39:41.125857: Epoch 837 
+2024-09-11 01:39:41.126502: Current learning rate: 0.00195 
+2024-09-11 01:44:06.644764: train_loss -0.762 
+2024-09-11 01:44:06.672940: val_loss -0.6976 
+2024-09-11 01:44:06.674176: Pseudo dice [0.9667, 0.9599, 0.9806, 0.9507, 0.8867, 0.9764, 0.9919] 
+2024-09-11 01:44:06.675440: Epoch time: 265.52 s 
+2024-09-11 01:44:08.985332:  
+2024-09-11 01:44:08.986197: Epoch 838 
+2024-09-11 01:44:08.986930: Current learning rate: 0.00194 
+2024-09-11 01:48:33.977267: train_loss -0.7584 
+2024-09-11 01:48:33.990197: val_loss -0.7567 
+2024-09-11 01:48:33.991079: Pseudo dice [0.9686, 0.9618, 0.9798, 0.9565, 0.895, 0.9752, 0.992] 
+2024-09-11 01:48:33.991814: Epoch time: 265.0 s 
+2024-09-11 01:48:33.992537: Yayy! New best EMA pseudo Dice: 0.9597 
+2024-09-11 01:48:39.463746:  
+2024-09-11 01:48:39.464823: Epoch 839 
+2024-09-11 01:48:39.465606: Current learning rate: 0.00193 
+2024-09-11 01:53:07.624941: train_loss -0.7705 
+2024-09-11 01:53:07.640293: val_loss -0.7624 
+2024-09-11 01:53:07.641051: Pseudo dice [0.9666, 0.9621, 0.9802, 0.9561, 0.8925, 0.9744, 0.9911] 
+2024-09-11 01:53:07.641663: Epoch time: 268.17 s 
+2024-09-11 01:53:07.642328: Yayy! New best EMA pseudo Dice: 0.9598 
+2024-09-11 01:53:13.341983:  
+2024-09-11 01:53:13.342878: Epoch 840 
+2024-09-11 01:53:13.343595: Current learning rate: 0.00192 
+2024-09-11 01:57:46.078647: train_loss -0.759 
+2024-09-11 01:57:46.080700: val_loss -0.8041 
+2024-09-11 01:57:46.081443: Pseudo dice [0.964, 0.9601, 0.9792, 0.9504, 0.893, 0.9747, 0.9913] 
+2024-09-11 01:57:46.082179: Epoch time: 272.74 s 
+2024-09-11 01:57:48.999862:  
+2024-09-11 01:57:49.000765: Epoch 841 
+2024-09-11 01:57:49.001435: Current learning rate: 0.00191 
+2024-09-11 02:02:15.709567: train_loss -0.7553 
+2024-09-11 02:02:15.712052: val_loss -0.7322 
+2024-09-11 02:02:15.713012: Pseudo dice [0.9647, 0.9593, 0.9776, 0.9602, 0.8859, 0.9713, 0.9904] 
+2024-09-11 02:02:15.713847: Epoch time: 266.71 s 
+2024-09-11 02:02:18.653764:  
+2024-09-11 02:02:18.654958: Epoch 842 
+2024-09-11 02:02:18.655755: Current learning rate: 0.0019 
+2024-09-11 02:06:34.146202: train_loss -0.7576 
+2024-09-11 02:06:34.148634: val_loss -0.7457 
+2024-09-11 02:06:34.149560: Pseudo dice [0.9664, 0.9597, 0.9791, 0.9568, 0.8879, 0.9733, 0.9909] 
+2024-09-11 02:06:34.150321: Epoch time: 255.5 s 
+2024-09-11 02:06:37.183732:  
+2024-09-11 02:06:37.184541: Epoch 843 
+2024-09-11 02:06:37.185289: Current learning rate: 0.00189 
+2024-09-11 02:10:54.876759: train_loss -0.7694 
+2024-09-11 02:10:54.890020: val_loss -0.7267 
+2024-09-11 02:10:54.890858: Pseudo dice [0.9665, 0.9608, 0.9802, 0.9543, 0.8682, 0.9729, 0.9887] 
+2024-09-11 02:10:54.891617: Epoch time: 257.7 s 
+2024-09-11 02:10:57.827165:  
+2024-09-11 02:10:57.828238: Epoch 844 
+2024-09-11 02:10:57.829010: Current learning rate: 0.00188 
+2024-09-11 02:15:18.272626: train_loss -0.7643 
+2024-09-11 02:15:18.290583: val_loss -0.7674 
+2024-09-11 02:15:18.291526: Pseudo dice [0.9668, 0.9606, 0.9814, 0.9589, 0.896, 0.9751, 0.9924] 
+2024-09-11 02:15:18.292289: Epoch time: 260.45 s 
+2024-09-11 02:15:21.945880:  
+2024-09-11 02:15:21.946803: Epoch 845 
+2024-09-11 02:15:21.947613: Current learning rate: 0.00187 
+2024-09-11 02:19:36.388237: train_loss -0.7456 
+2024-09-11 02:19:36.402101: val_loss -0.7392 
+2024-09-11 02:19:36.402983: Pseudo dice [0.9655, 0.9594, 0.9793, 0.9561, 0.8798, 0.9764, 0.9921] 
+2024-09-11 02:19:36.403761: Epoch time: 254.46 s 
+2024-09-11 02:19:39.253796:  
+2024-09-11 02:19:39.268707: Epoch 846 
+2024-09-11 02:19:39.269433: Current learning rate: 0.00186 
+2024-09-11 02:24:17.678725: train_loss -0.7625 
+2024-09-11 02:24:17.688502: val_loss -0.7284 
+2024-09-11 02:24:17.689395: Pseudo dice [0.965, 0.9618, 0.9772, 0.9411, 0.8855, 0.9698, 0.9916] 
+2024-09-11 02:24:17.690179: Epoch time: 278.43 s 
+2024-09-11 02:24:20.402789:  
+2024-09-11 02:24:20.407670: Epoch 847 
+2024-09-11 02:24:20.408511: Current learning rate: 0.00185 
+2024-09-11 02:28:52.005735: train_loss -0.7636 
+2024-09-11 02:28:52.007968: val_loss -0.7493 
+2024-09-11 02:28:52.008761: Pseudo dice [0.9678, 0.9611, 0.9802, 0.958, 0.8914, 0.9752, 0.9922] 
+2024-09-11 02:28:52.009514: Epoch time: 271.61 s 
+2024-09-11 02:28:54.186046:  
+2024-09-11 02:28:54.196174: Epoch 848 
+2024-09-11 02:28:54.197049: Current learning rate: 0.00184 
+2024-09-11 02:33:05.325280: train_loss -0.7527 
+2024-09-11 02:33:05.338658: val_loss -0.7935 
+2024-09-11 02:33:05.339464: Pseudo dice [0.9665, 0.9599, 0.9813, 0.9559, 0.8958, 0.9775, 0.9934] 
+2024-09-11 02:33:05.340177: Epoch time: 251.14 s 
+2024-09-11 02:33:08.971378:  
+2024-09-11 02:33:08.972382: Epoch 849 
+2024-09-11 02:33:08.973099: Current learning rate: 0.00182 
+2024-09-11 02:37:30.393058: train_loss -0.7724 
+2024-09-11 02:37:30.395226: val_loss -0.7681 
+2024-09-11 02:37:30.395941: Pseudo dice [0.9679, 0.9612, 0.9807, 0.9568, 0.8939, 0.9762, 0.9927] 
+2024-09-11 02:37:30.396704: Epoch time: 261.43 s 
+2024-09-11 02:37:37.791873:  
+2024-09-11 02:37:37.794857: Epoch 850 
+2024-09-11 02:37:37.795637: Current learning rate: 0.00181 
+2024-09-11 02:42:08.446002: train_loss -0.7647 
+2024-09-11 02:42:08.453862: val_loss -0.7788 
+2024-09-11 02:42:08.454680: Pseudo dice [0.9681, 0.9592, 0.9818, 0.9536, 0.8893, 0.9715, 0.9923] 
+2024-09-11 02:42:08.455285: Epoch time: 270.66 s 
+2024-09-11 02:42:12.224164:  
+2024-09-11 02:42:12.240108: Epoch 851 
+2024-09-11 02:42:12.240900: Current learning rate: 0.0018 
+2024-09-11 02:46:37.253752: train_loss -0.7726 
+2024-09-11 02:46:37.270374: val_loss -0.7973 
+2024-09-11 02:46:37.271279: Pseudo dice [0.9694, 0.9611, 0.9816, 0.9495, 0.8895, 0.9747, 0.9921] 
+2024-09-11 02:46:37.272125: Epoch time: 265.03 s 
+2024-09-11 02:46:40.742684:  
+2024-09-11 02:46:40.743560: Epoch 852 
+2024-09-11 02:46:40.744322: Current learning rate: 0.00179 
+2024-09-11 02:51:05.109256: train_loss -0.7521 
+2024-09-11 02:51:05.111487: val_loss -0.7101 
+2024-09-11 02:51:05.112178: Pseudo dice [0.9616, 0.9582, 0.977, 0.9514, 0.8771, 0.9721, 0.9899] 
+2024-09-11 02:51:05.112866: Epoch time: 264.37 s 
+2024-09-11 02:51:07.321008:  
+2024-09-11 02:51:07.321949: Epoch 853 
+2024-09-11 02:51:07.322582: Current learning rate: 0.00178 
+2024-09-11 02:55:36.726450: train_loss -0.7694 
+2024-09-11 02:55:36.742529: val_loss -0.7372 
+2024-09-11 02:55:36.743530: Pseudo dice [0.9623, 0.9607, 0.9742, 0.9516, 0.8881, 0.9746, 0.9907] 
+2024-09-11 02:55:36.744367: Epoch time: 269.41 s 
+2024-09-11 02:55:39.492322:  
+2024-09-11 02:55:39.508791: Epoch 854 
+2024-09-11 02:55:39.509676: Current learning rate: 0.00177 
+2024-09-11 03:00:09.226566: train_loss -0.7486 
+2024-09-11 03:00:09.228781: val_loss -0.7387 
+2024-09-11 03:00:09.229525: Pseudo dice [0.9684, 0.9622, 0.9823, 0.9606, 0.8968, 0.9745, 0.9915] 
+2024-09-11 03:00:09.230401: Epoch time: 269.74 s 
+2024-09-11 03:00:11.653191:  
+2024-09-11 03:00:11.654074: Epoch 855 
+2024-09-11 03:00:11.654674: Current learning rate: 0.00176 
+2024-09-11 03:04:27.206041: train_loss -0.7486 
+2024-09-11 03:04:27.208305: val_loss -0.7813 
+2024-09-11 03:04:27.209079: Pseudo dice [0.9675, 0.9599, 0.9775, 0.954, 0.8885, 0.976, 0.9921] 
+2024-09-11 03:04:27.209796: Epoch time: 255.56 s 
+2024-09-11 03:04:30.226479:  
+2024-09-11 03:04:30.240646: Epoch 856 
+2024-09-11 03:04:30.241415: Current learning rate: 0.00175 
+2024-09-11 03:08:58.989056: train_loss -0.7635 
+2024-09-11 03:08:58.991685: val_loss -0.7473 
+2024-09-11 03:08:58.992738: Pseudo dice [0.9659, 0.9618, 0.9768, 0.9549, 0.8863, 0.9728, 0.9916] 
+2024-09-11 03:08:58.993707: Epoch time: 268.77 s 
+2024-09-11 03:09:02.079131:  
+2024-09-11 03:09:02.080602: Epoch 857 
+2024-09-11 03:09:02.081459: Current learning rate: 0.00174 
+2024-09-11 03:13:41.571193: train_loss -0.7762 
+2024-09-11 03:13:41.589794: val_loss -0.7866 
+2024-09-11 03:13:41.590537: Pseudo dice [0.97, 0.9613, 0.9839, 0.9506, 0.8979, 0.9738, 0.9917] 
+2024-09-11 03:13:41.591188: Epoch time: 279.5 s 
+2024-09-11 03:13:45.527460:  
+2024-09-11 03:13:45.528339: Epoch 858 
+2024-09-11 03:13:45.529031: Current learning rate: 0.00173 
+2024-09-11 03:18:11.013948: train_loss -0.7628 
+2024-09-11 03:18:11.030043: val_loss -0.7376 
+2024-09-11 03:18:11.030862: Pseudo dice [0.9638, 0.9577, 0.9783, 0.9485, 0.886, 0.9724, 0.9927] 
+2024-09-11 03:18:11.031590: Epoch time: 265.49 s 
+2024-09-11 03:18:13.978278:  
+2024-09-11 03:18:13.992662: Epoch 859 
+2024-09-11 03:18:13.993527: Current learning rate: 0.00172 
+2024-09-11 03:22:42.005041: train_loss -0.744 
+2024-09-11 03:22:42.008622: val_loss -0.7922 
+2024-09-11 03:22:42.015064: Pseudo dice [0.963, 0.9608, 0.9772, 0.9548, 0.8897, 0.9751, 0.9918] 
+2024-09-11 03:22:42.016419: Epoch time: 268.03 s 
+2024-09-11 03:22:46.108822:  
+2024-09-11 03:22:46.109671: Epoch 860 
+2024-09-11 03:22:46.110353: Current learning rate: 0.0017 
+2024-09-11 03:26:58.065907: train_loss -0.7609 
+2024-09-11 03:26:58.079335: val_loss -0.77 
+2024-09-11 03:26:58.080037: Pseudo dice [0.9656, 0.9596, 0.9802, 0.95, 0.8917, 0.9758, 0.993] 
+2024-09-11 03:26:58.080784: Epoch time: 251.96 s 
+2024-09-11 03:27:00.519470:  
+2024-09-11 03:27:00.524029: Epoch 861 
+2024-09-11 03:27:00.537033: Current learning rate: 0.00169 
+2024-09-11 03:31:35.347943: train_loss -0.759 
+2024-09-11 03:31:35.366052: val_loss -0.7955 
+2024-09-11 03:31:35.366789: Pseudo dice [0.9702, 0.9621, 0.9836, 0.9618, 0.8998, 0.9772, 0.9932] 
+2024-09-11 03:31:35.367437: Epoch time: 274.83 s 
+2024-09-11 03:31:37.920418:  
+2024-09-11 03:31:37.921955: Epoch 862 
+2024-09-11 03:31:37.923147: Current learning rate: 0.00168 
+2024-09-11 03:35:49.910747: train_loss -0.765 
+2024-09-11 03:35:49.937979: val_loss -0.7631 
+2024-09-11 03:35:49.938720: Pseudo dice [0.9666, 0.9608, 0.9805, 0.9571, 0.8925, 0.9776, 0.9922] 
+2024-09-11 03:35:49.939400: Epoch time: 252.0 s 
+2024-09-11 03:35:49.939950: Yayy! New best EMA pseudo Dice: 0.9598 
+2024-09-11 03:35:55.395312:  
+2024-09-11 03:35:55.396288: Epoch 863 
+2024-09-11 03:35:55.397057: Current learning rate: 0.00167 
+2024-09-11 03:40:26.724904: train_loss -0.7702 
+2024-09-11 03:40:26.727446: val_loss -0.7405 
+2024-09-11 03:40:26.728662: Pseudo dice [0.9657, 0.9606, 0.9816, 0.9491, 0.8918, 0.9744, 0.9904] 
+2024-09-11 03:40:26.729887: Epoch time: 271.33 s 
+2024-09-11 03:40:29.693026:  
+2024-09-11 03:40:29.694131: Epoch 864 
+2024-09-11 03:40:29.694977: Current learning rate: 0.00166 
+2024-09-11 03:44:37.001080: train_loss -0.7685 
+2024-09-11 03:44:37.003372: val_loss -0.7129 
+2024-09-11 03:44:37.004104: Pseudo dice [0.9652, 0.9623, 0.9786, 0.9561, 0.8933, 0.9763, 0.9921] 
+2024-09-11 03:44:37.004807: Epoch time: 247.31 s 
+2024-09-11 03:44:37.005462: Yayy! New best EMA pseudo Dice: 0.9598 
+2024-09-11 03:44:42.142283:  
+2024-09-11 03:44:42.143064: Epoch 865 
+2024-09-11 03:44:42.143675: Current learning rate: 0.00165 
+2024-09-11 03:49:12.523652: train_loss -0.76 
+2024-09-11 03:49:12.525864: val_loss -0.7562 
+2024-09-11 03:49:12.526850: Pseudo dice [0.9683, 0.9626, 0.9816, 0.9537, 0.8957, 0.9764, 0.9931] 
+2024-09-11 03:49:12.527623: Epoch time: 270.38 s 
+2024-09-11 03:49:12.528246: Yayy! New best EMA pseudo Dice: 0.96 
+2024-09-11 03:49:18.032412:  
+2024-09-11 03:49:18.033370: Epoch 866 
+2024-09-11 03:49:18.034116: Current learning rate: 0.00164 
+2024-09-11 03:53:38.886426: train_loss -0.7678 
+2024-09-11 03:53:38.918139: val_loss -0.7416 
+2024-09-11 03:53:38.919213: Pseudo dice [0.9696, 0.9619, 0.9828, 0.958, 0.8955, 0.978, 0.9923] 
+2024-09-11 03:53:38.920664: Epoch time: 260.86 s 
+2024-09-11 03:53:38.921835: Yayy! New best EMA pseudo Dice: 0.9603 
+2024-09-11 03:53:44.866935:  
+2024-09-11 03:53:44.871603: Epoch 867 
+2024-09-11 03:53:44.872360: Current learning rate: 0.00163 
+2024-09-11 03:58:14.548129: train_loss -0.7427 
+2024-09-11 03:58:14.550487: val_loss -0.7713 
+2024-09-11 03:58:14.551311: Pseudo dice [0.9686, 0.9644, 0.9813, 0.956, 0.8948, 0.9758, 0.9935] 
+2024-09-11 03:58:14.551912: Epoch time: 269.69 s 
+2024-09-11 03:58:14.552524: Yayy! New best EMA pseudo Dice: 0.9604 
+2024-09-11 03:58:20.484246:  
+2024-09-11 03:58:20.485253: Epoch 868 
+2024-09-11 03:58:20.486063: Current learning rate: 0.00162 
+2024-09-11 04:02:56.362588: train_loss -0.7588 
+2024-09-11 04:02:56.394077: val_loss -0.7824 
+2024-09-11 04:02:56.394936: Pseudo dice [0.9675, 0.9588, 0.982, 0.9568, 0.8882, 0.972, 0.9912] 
+2024-09-11 04:02:56.395673: Epoch time: 275.88 s 
+2024-09-11 04:02:59.287622:  
+2024-09-11 04:02:59.288436: Epoch 869 
+2024-09-11 04:02:59.289008: Current learning rate: 0.00161 
+2024-09-11 04:07:17.303384: train_loss -0.7519 
+2024-09-11 04:07:17.313398: val_loss -0.7283 
+2024-09-11 04:07:17.314484: Pseudo dice [0.9667, 0.9594, 0.9791, 0.9511, 0.8862, 0.9763, 0.9925] 
+2024-09-11 04:07:17.315239: Epoch time: 258.02 s 
+2024-09-11 04:07:21.348095:  
+2024-09-11 04:07:21.349435: Epoch 870 
+2024-09-11 04:07:21.350636: Current learning rate: 0.00159 
+2024-09-11 04:11:52.687038: train_loss -0.7302 
+2024-09-11 04:11:52.708845: val_loss -0.7473 
+2024-09-11 04:11:52.709603: Pseudo dice [0.968, 0.9625, 0.981, 0.9585, 0.9006, 0.9753, 0.9919] 
+2024-09-11 04:11:52.710265: Epoch time: 271.36 s 
+2024-09-11 04:11:55.340812:  
+2024-09-11 04:11:55.341740: Epoch 871 
+2024-09-11 04:11:55.342424: Current learning rate: 0.00158 
+2024-09-11 04:16:02.635607: train_loss -0.7545 
+2024-09-11 04:16:02.646049: val_loss -0.7744 
+2024-09-11 04:16:02.646826: Pseudo dice [0.9679, 0.9593, 0.9809, 0.9574, 0.8908, 0.9765, 0.9921] 
+2024-09-11 04:16:02.647488: Epoch time: 247.3 s 
+2024-09-11 04:16:02.648350: Yayy! New best EMA pseudo Dice: 0.9605 
+2024-09-11 04:16:07.917110:  
+2024-09-11 04:16:07.918044: Epoch 872 
+2024-09-11 04:16:07.918763: Current learning rate: 0.00157 
+2024-09-11 04:20:17.071277: train_loss -0.7405 
+2024-09-11 04:20:17.085047: val_loss -0.7502 
+2024-09-11 04:20:17.085758: Pseudo dice [0.9685, 0.9623, 0.9808, 0.9505, 0.8909, 0.9739, 0.9922] 
+2024-09-11 04:20:17.086428: Epoch time: 249.16 s 
+2024-09-11 04:20:21.573069:  
+2024-09-11 04:20:21.574282: Epoch 873 
+2024-09-11 04:20:21.575067: Current learning rate: 0.00156 
+2024-09-11 04:24:20.389166: train_loss -0.7606 
+2024-09-11 04:24:20.390934: val_loss -0.7528 
+2024-09-11 04:24:20.392318: Pseudo dice [0.9662, 0.9606, 0.9783, 0.9519, 0.887, 0.9744, 0.9911] 
+2024-09-11 04:24:20.393430: Epoch time: 238.82 s 
+2024-09-11 04:24:23.916157:  
+2024-09-11 04:24:23.917610: Epoch 874 
+2024-09-11 04:24:23.918945: Current learning rate: 0.00155 
+2024-09-11 04:28:45.324992: train_loss -0.7576 
+2024-09-11 04:28:45.326374: val_loss -0.7606 
+2024-09-11 04:28:45.327129: Pseudo dice [0.9689, 0.9595, 0.9822, 0.9534, 0.8809, 0.9755, 0.9921] 
+2024-09-11 04:28:45.327755: Epoch time: 261.43 s 
+2024-09-11 04:28:48.292748:  
+2024-09-11 04:28:48.293686: Epoch 875 
+2024-09-11 04:28:48.294235: Current learning rate: 0.00154 
+2024-09-11 04:33:01.380162: train_loss -0.7671 
+2024-09-11 04:33:01.397216: val_loss -0.756 
+2024-09-11 04:33:01.398138: Pseudo dice [0.9691, 0.9634, 0.9821, 0.954, 0.8997, 0.9778, 0.9925] 
+2024-09-11 04:33:01.398975: Epoch time: 253.09 s 
+2024-09-11 04:33:04.120488:  
+2024-09-11 04:33:04.121372: Epoch 876 
+2024-09-11 04:33:04.122044: Current learning rate: 0.00153 
+2024-09-11 04:37:15.234803: train_loss -0.7742 
+2024-09-11 04:37:15.235819: val_loss -0.7649 
+2024-09-11 04:37:15.248218: Pseudo dice [0.9684, 0.96, 0.9806, 0.9565, 0.8871, 0.9763, 0.9912] 
+2024-09-11 04:37:15.249216: Epoch time: 251.12 s 
+2024-09-11 04:37:18.081259:  
+2024-09-11 04:37:18.082445: Epoch 877 
+2024-09-11 04:37:18.083130: Current learning rate: 0.00152 
+2024-09-11 04:41:22.346179: train_loss -0.7701 
+2024-09-11 04:41:22.347446: val_loss -0.7437 
+2024-09-11 04:41:22.350510: Pseudo dice [0.9685, 0.9625, 0.9815, 0.9544, 0.898, 0.9774, 0.9917] 
+2024-09-11 04:41:22.364756: Epoch time: 244.27 s 
+2024-09-11 04:41:22.365486: Yayy! New best EMA pseudo Dice: 0.9605 
+2024-09-11 04:41:28.729218:  
+2024-09-11 04:41:28.730129: Epoch 878 
+2024-09-11 04:41:28.730807: Current learning rate: 0.00151 
+2024-09-11 04:45:34.012250: train_loss -0.7565 
+2024-09-11 04:45:34.013546: val_loss -0.7427 
+2024-09-11 04:45:34.014422: Pseudo dice [0.9675, 0.961, 0.981, 0.9561, 0.8844, 0.9738, 0.9903] 
+2024-09-11 04:45:34.015286: Epoch time: 245.29 s 
+2024-09-11 04:45:36.484837:  
+2024-09-11 04:45:36.486164: Epoch 879 
+2024-09-11 04:45:36.487020: Current learning rate: 0.00149 
+2024-09-11 04:49:49.751562: train_loss -0.7458 
+2024-09-11 04:49:49.765218: val_loss -0.7581 
+2024-09-11 04:49:49.766078: Pseudo dice [0.9655, 0.9597, 0.98, 0.9468, 0.8937, 0.9742, 0.9915] 
+2024-09-11 04:49:49.766789: Epoch time: 253.27 s 
+2024-09-11 04:49:53.218366:  
+2024-09-11 04:49:53.219226: Epoch 880 
+2024-09-11 04:49:53.219814: Current learning rate: 0.00148 
+2024-09-11 04:54:20.895023: train_loss -0.758 
+2024-09-11 04:54:20.897418: val_loss -0.7746 
+2024-09-11 04:54:20.898278: Pseudo dice [0.9685, 0.9601, 0.9832, 0.9569, 0.8968, 0.9761, 0.9925] 
+2024-09-11 04:54:20.899077: Epoch time: 267.68 s 
+2024-09-11 04:54:23.556073:  
+2024-09-11 04:54:23.580591: Epoch 881 
+2024-09-11 04:54:23.581526: Current learning rate: 0.00147 
+2024-09-11 04:58:40.450597: train_loss -0.7614 
+2024-09-11 04:58:40.453625: val_loss -0.7756 
+2024-09-11 04:58:40.454965: Pseudo dice [0.9681, 0.9602, 0.9809, 0.956, 0.8944, 0.9772, 0.9922] 
+2024-09-11 04:58:40.456208: Epoch time: 256.9 s 
+2024-09-11 04:58:43.325846:  
+2024-09-11 04:58:43.336594: Epoch 882 
+2024-09-11 04:58:43.337345: Current learning rate: 0.00146 
+2024-09-11 05:02:58.219018: train_loss -0.7589 
+2024-09-11 05:02:58.242656: val_loss -0.7543 
+2024-09-11 05:02:58.243536: Pseudo dice [0.9686, 0.9612, 0.9827, 0.9497, 0.8912, 0.9746, 0.9922] 
+2024-09-11 05:02:58.244544: Epoch time: 254.9 s 
+2024-09-11 05:03:00.392754:  
+2024-09-11 05:03:00.402297: Epoch 883 
+2024-09-11 05:03:00.403168: Current learning rate: 0.00145 
+2024-09-11 05:07:19.963916: train_loss -0.7479 
+2024-09-11 05:07:19.966211: val_loss -0.7509 
+2024-09-11 05:07:19.967130: Pseudo dice [0.9681, 0.9587, 0.9801, 0.9543, 0.8828, 0.9747, 0.9927] 
+2024-09-11 05:07:19.967874: Epoch time: 259.58 s 
+2024-09-11 05:07:22.132529:  
+2024-09-11 05:07:22.133702: Epoch 884 
+2024-09-11 05:07:22.134374: Current learning rate: 0.00144 
+2024-09-11 05:11:27.335818: train_loss -0.7619 
+2024-09-11 05:11:27.354020: val_loss -0.8241 
+2024-09-11 05:11:27.354808: Pseudo dice [0.9689, 0.9597, 0.9815, 0.9525, 0.8969, 0.9766, 0.9936] 
+2024-09-11 05:11:27.355473: Epoch time: 245.21 s 
+2024-09-11 05:11:29.625394:  
+2024-09-11 05:11:29.626307: Epoch 885 
+2024-09-11 05:11:29.627098: Current learning rate: 0.00143 
+2024-09-11 05:15:36.151596: train_loss -0.7473 
+2024-09-11 05:15:36.173859: val_loss -0.7394 
+2024-09-11 05:15:36.174771: Pseudo dice [0.9685, 0.9626, 0.9815, 0.9567, 0.8855, 0.9766, 0.99] 
+2024-09-11 05:15:36.175458: Epoch time: 246.53 s 
+2024-09-11 05:15:39.299651:  
+2024-09-11 05:15:39.312074: Epoch 886 
+2024-09-11 05:15:39.313025: Current learning rate: 0.00142 
+2024-09-11 05:20:01.084349: train_loss -0.7554 
+2024-09-11 05:20:01.101389: val_loss -0.7431 
+2024-09-11 05:20:01.102286: Pseudo dice [0.9683, 0.9614, 0.9813, 0.9529, 0.892, 0.9761, 0.9922] 
+2024-09-11 05:20:01.103225: Epoch time: 261.79 s 
+2024-09-11 05:20:03.792285:  
+2024-09-11 05:20:03.793142: Epoch 887 
+2024-09-11 05:20:03.793779: Current learning rate: 0.00141 
+2024-09-11 05:24:24.810841: train_loss -0.7684 
+2024-09-11 05:24:24.831241: val_loss -0.7639 
+2024-09-11 05:24:24.832231: Pseudo dice [0.9664, 0.9605, 0.98, 0.9516, 0.8821, 0.9712, 0.9911] 
+2024-09-11 05:24:24.833243: Epoch time: 261.02 s 
+2024-09-11 05:24:27.792157:  
+2024-09-11 05:24:27.793046: Epoch 888 
+2024-09-11 05:24:27.793765: Current learning rate: 0.00139 
+2024-09-11 05:28:29.793407: train_loss -0.7608 
+2024-09-11 05:28:29.806039: val_loss -0.7496 
+2024-09-11 05:28:29.806935: Pseudo dice [0.9679, 0.9623, 0.9814, 0.9541, 0.893, 0.9747, 0.9918] 
+2024-09-11 05:28:29.807881: Epoch time: 242.01 s 
+2024-09-11 05:28:32.475602:  
+2024-09-11 05:28:32.476922: Epoch 889 
+2024-09-11 05:28:32.478189: Current learning rate: 0.00138 
+2024-09-11 05:33:03.988381: train_loss -0.7645 
+2024-09-11 05:33:03.990689: val_loss -0.7824 
+2024-09-11 05:33:03.991505: Pseudo dice [0.9684, 0.9613, 0.9833, 0.9554, 0.8963, 0.9759, 0.9914] 
+2024-09-11 05:33:03.992284: Epoch time: 271.52 s 
+2024-09-11 05:33:07.073079:  
+2024-09-11 05:33:07.073940: Epoch 890 
+2024-09-11 05:33:07.074616: Current learning rate: 0.00137 
+2024-09-11 05:37:28.390971: train_loss -0.7613 
+2024-09-11 05:37:28.403391: val_loss -0.7328 
+2024-09-11 05:37:28.404361: Pseudo dice [0.9687, 0.9621, 0.9815, 0.9549, 0.8729, 0.974, 0.991] 
+2024-09-11 05:37:28.405186: Epoch time: 261.32 s 
+2024-09-11 05:37:31.857012:  
+2024-09-11 05:37:31.857840: Epoch 891 
+2024-09-11 05:37:31.858470: Current learning rate: 0.00136 
+2024-09-11 05:41:46.655240: train_loss -0.7545 
+2024-09-11 05:41:46.673676: val_loss -0.7423 
+2024-09-11 05:41:46.674573: Pseudo dice [0.9614, 0.9564, 0.9755, 0.9429, 0.8811, 0.9716, 0.9902] 
+2024-09-11 05:41:46.675406: Epoch time: 254.8 s 
+2024-09-11 05:41:48.967540:  
+2024-09-11 05:41:48.968874: Epoch 892 
+2024-09-11 05:41:48.969960: Current learning rate: 0.00135 
+2024-09-11 05:46:11.654995: train_loss -0.7626 
+2024-09-11 05:46:11.685592: val_loss -0.751 
+2024-09-11 05:46:11.686354: Pseudo dice [0.9653, 0.9596, 0.979, 0.9532, 0.8942, 0.9761, 0.9928] 
+2024-09-11 05:46:11.687067: Epoch time: 262.69 s 
+2024-09-11 05:46:14.816617:  
+2024-09-11 05:46:14.817649: Epoch 893 
+2024-09-11 05:46:14.818458: Current learning rate: 0.00134 
+2024-09-11 05:50:13.755594: train_loss -0.7554 
+2024-09-11 05:50:13.782176: val_loss -0.7691 
+2024-09-11 05:50:13.782865: Pseudo dice [0.9698, 0.9637, 0.9827, 0.9543, 0.8927, 0.9728, 0.9913] 
+2024-09-11 05:50:13.783561: Epoch time: 238.94 s 
+2024-09-11 05:50:16.813588:  
+2024-09-11 05:50:16.824711: Epoch 894 
+2024-09-11 05:50:16.825495: Current learning rate: 0.00133 
+2024-09-11 05:54:38.600609: train_loss -0.7548 
+2024-09-11 05:54:38.603007: val_loss -0.7707 
+2024-09-11 05:54:38.603765: Pseudo dice [0.9682, 0.9617, 0.9804, 0.9551, 0.8909, 0.9766, 0.992] 
+2024-09-11 05:54:38.604448: Epoch time: 261.79 s 
+2024-09-11 05:54:41.329116:  
+2024-09-11 05:54:41.344957: Epoch 895 
+2024-09-11 05:54:41.345931: Current learning rate: 0.00132 
+2024-09-11 05:58:52.269751: train_loss -0.7552 
+2024-09-11 05:58:52.286211: val_loss -0.7841 
+2024-09-11 05:58:52.287031: Pseudo dice [0.9667, 0.9618, 0.982, 0.9578, 0.898, 0.9776, 0.9927] 
+2024-09-11 05:58:52.287676: Epoch time: 250.94 s 
+2024-09-11 05:58:55.846067:  
+2024-09-11 05:58:55.847193: Epoch 896 
+2024-09-11 05:58:55.848145: Current learning rate: 0.0013 
+2024-09-11 06:03:10.694668: train_loss -0.7518 
+2024-09-11 06:03:10.710068: val_loss -0.7629 
+2024-09-11 06:03:10.710861: Pseudo dice [0.9666, 0.9607, 0.9805, 0.9528, 0.896, 0.9754, 0.9918] 
+2024-09-11 06:03:10.711624: Epoch time: 254.85 s 
+2024-09-11 06:03:13.161616:  
+2024-09-11 06:03:13.176764: Epoch 897 
+2024-09-11 06:03:13.177775: Current learning rate: 0.00129 
+2024-09-11 06:07:48.310026: train_loss -0.7573 
+2024-09-11 06:07:48.312304: val_loss -0.7849 
+2024-09-11 06:07:48.312991: Pseudo dice [0.9672, 0.9598, 0.9824, 0.9549, 0.8908, 0.9758, 0.9919] 
+2024-09-11 06:07:48.313711: Epoch time: 275.15 s 
+2024-09-11 06:07:52.497692:  
+2024-09-11 06:07:52.498563: Epoch 898 
+2024-09-11 06:07:52.499333: Current learning rate: 0.00128 
+2024-09-11 06:12:13.669310: train_loss -0.7544 
+2024-09-11 06:12:13.671950: val_loss -0.753 
+2024-09-11 06:12:13.672985: Pseudo dice [0.9683, 0.9626, 0.9814, 0.9592, 0.8961, 0.9761, 0.9908] 
+2024-09-11 06:12:13.673842: Epoch time: 261.18 s 
+2024-09-11 06:12:15.848286:  
+2024-09-11 06:12:15.857540: Epoch 899 
+2024-09-11 06:12:15.858582: Current learning rate: 0.00127 
+2024-09-11 06:16:34.712342: train_loss -0.7635 
+2024-09-11 06:16:34.714981: val_loss -0.7749 
+2024-09-11 06:16:34.715840: Pseudo dice [0.9702, 0.9642, 0.9829, 0.958, 0.8902, 0.9753, 0.9917] 
+2024-09-11 06:16:34.716564: Epoch time: 258.85 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_11_06_42_34.txt

@@ -0,0 +1,745 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-11 06:42:43.955588: Using torch.compile... 
+2024-09-11 06:42:58.104252: do_dummy_2d_data_aug: False 
+2024-09-11 06:42:58.110365: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-11 06:42:58.112206: The split file contains 5 splits. 
+2024-09-11 06:42:58.112992: Desired fold for training: 0 
+2024-09-11 06:42:58.113758: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-11 06:43:09.345969: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-11 06:43:20.669152:  
+2024-09-11 06:43:20.680806: Epoch 900 
+2024-09-11 06:43:20.681933: Current learning rate: 0.00126 
+2024-09-11 06:56:41.016367: train_loss -0.7805 
+2024-09-11 06:56:41.018920: val_loss -0.7607 
+2024-09-11 06:56:41.020010: Pseudo dice [0.9677, 0.9624, 0.9797, 0.9553, 0.8819, 0.9763, 0.9913] 
+2024-09-11 06:56:41.020890: Epoch time: 800.35 s 
+2024-09-11 06:56:42.531756:  
+2024-09-11 06:56:42.532915: Epoch 901 
+2024-09-11 06:56:42.533906: Current learning rate: 0.00125 
+2024-09-11 07:00:42.477643: train_loss -0.761 
+2024-09-11 07:00:42.497912: val_loss -0.78 
+2024-09-11 07:00:42.498853: Pseudo dice [0.9678, 0.9592, 0.9817, 0.9567, 0.8943, 0.9771, 0.9916] 
+2024-09-11 07:00:42.499707: Epoch time: 239.95 s 
+2024-09-11 07:00:45.211778:  
+2024-09-11 07:00:45.212790: Epoch 902 
+2024-09-11 07:00:45.213735: Current learning rate: 0.00124 
+2024-09-11 07:04:53.324517: train_loss -0.752 
+2024-09-11 07:04:53.345953: val_loss -0.7729 
+2024-09-11 07:04:53.347249: Pseudo dice [0.9701, 0.9633, 0.9809, 0.9551, 0.8891, 0.9754, 0.9927] 
+2024-09-11 07:04:53.348390: Epoch time: 248.12 s 
+2024-09-11 07:04:53.349449: Yayy! New best EMA pseudo Dice: 0.9605 
+2024-09-11 07:05:01.124498:  
+2024-09-11 07:05:01.125554: Epoch 903 
+2024-09-11 07:05:01.126439: Current learning rate: 0.00122 
+2024-09-11 07:09:12.109399: train_loss -0.7659 
+2024-09-11 07:09:12.111792: val_loss -0.7752 
+2024-09-11 07:09:12.112767: Pseudo dice [0.9677, 0.9582, 0.9818, 0.9536, 0.8932, 0.9782, 0.9925] 
+2024-09-11 07:09:12.113844: Epoch time: 250.99 s 
+2024-09-11 07:09:12.114812: Yayy! New best EMA pseudo Dice: 0.9605 
+2024-09-11 07:09:17.170037:  
+2024-09-11 07:09:17.171251: Epoch 904 
+2024-09-11 07:09:17.172101: Current learning rate: 0.00121 
+2024-09-11 07:13:30.619256: train_loss -0.7726 
+2024-09-11 07:13:30.633017: val_loss -0.7422 
+2024-09-11 07:13:30.634171: Pseudo dice [0.9686, 0.9615, 0.9821, 0.9559, 0.8939, 0.9764, 0.9919] 
+2024-09-11 07:13:30.635428: Epoch time: 253.45 s 
+2024-09-11 07:13:30.636450: Yayy! New best EMA pseudo Dice: 0.9606 
+2024-09-11 07:13:35.943038:  
+2024-09-11 07:13:35.944103: Epoch 905 
+2024-09-11 07:13:35.944946: Current learning rate: 0.0012 
+2024-09-11 07:17:34.317254: train_loss -0.7668 
+2024-09-11 07:17:34.323313: val_loss -0.7795 
+2024-09-11 07:17:34.324394: Pseudo dice [0.9684, 0.962, 0.9825, 0.9561, 0.898, 0.9756, 0.9919] 
+2024-09-11 07:17:34.325468: Epoch time: 238.38 s 
+2024-09-11 07:17:34.326455: Yayy! New best EMA pseudo Dice: 0.9607 
+2024-09-11 07:17:38.343637:  
+2024-09-11 07:17:38.344640: Epoch 906 
+2024-09-11 07:17:38.345498: Current learning rate: 0.00119 
+2024-09-11 07:21:43.696677: train_loss -0.7521 
+2024-09-11 07:21:43.718045: val_loss -0.7603 
+2024-09-11 07:21:43.719106: Pseudo dice [0.9705, 0.9622, 0.9824, 0.9564, 0.894, 0.978, 0.9934] 
+2024-09-11 07:21:43.720222: Epoch time: 245.36 s 
+2024-09-11 07:21:43.721218: Yayy! New best EMA pseudo Dice: 0.9609 
+2024-09-11 07:21:49.551921:  
+2024-09-11 07:21:49.553099: Epoch 907 
+2024-09-11 07:21:49.553987: Current learning rate: 0.00118 
+2024-09-11 07:26:07.068816: train_loss -0.7597 
+2024-09-11 07:26:07.086238: val_loss -0.7502 
+2024-09-11 07:26:07.087224: Pseudo dice [0.9668, 0.9618, 0.9789, 0.9555, 0.8882, 0.9767, 0.9911] 
+2024-09-11 07:26:07.088150: Epoch time: 257.52 s 
+2024-09-11 07:26:10.465950:  
+2024-09-11 07:26:10.480926: Epoch 908 
+2024-09-11 07:26:10.482188: Current learning rate: 0.00117 
+2024-09-11 07:30:30.942825: train_loss -0.7598 
+2024-09-11 07:30:30.945637: val_loss -0.7903 
+2024-09-11 07:30:30.946789: Pseudo dice [0.9692, 0.9616, 0.9828, 0.9552, 0.8889, 0.9751, 0.9919] 
+2024-09-11 07:30:30.948109: Epoch time: 260.48 s 
+2024-09-11 07:30:33.169305:  
+2024-09-11 07:30:33.170758: Epoch 909 
+2024-09-11 07:30:33.172656: Current learning rate: 0.00116 
+2024-09-11 07:34:47.436793: train_loss -0.7589 
+2024-09-11 07:34:47.454203: val_loss -0.7652 
+2024-09-11 07:34:47.455263: Pseudo dice [0.97, 0.962, 0.9834, 0.9543, 0.8847, 0.9772, 0.9919] 
+2024-09-11 07:34:47.456154: Epoch time: 254.28 s 
+2024-09-11 07:34:50.084702:  
+2024-09-11 07:34:50.100896: Epoch 910 
+2024-09-11 07:34:50.101998: Current learning rate: 0.00115 
+2024-09-11 07:39:06.096661: train_loss -0.7615 
+2024-09-11 07:39:06.118515: val_loss -0.7507 
+2024-09-11 07:39:06.119525: Pseudo dice [0.9695, 0.9606, 0.9825, 0.9512, 0.8902, 0.9751, 0.9903] 
+2024-09-11 07:39:06.120591: Epoch time: 256.02 s 
+2024-09-11 07:39:09.414688:  
+2024-09-11 07:39:09.429099: Epoch 911 
+2024-09-11 07:39:09.430174: Current learning rate: 0.00113 
+2024-09-11 07:43:35.656014: train_loss -0.7641 
+2024-09-11 07:43:35.670226: val_loss -0.7829 
+2024-09-11 07:43:35.671210: Pseudo dice [0.971, 0.9616, 0.9839, 0.9587, 0.9023, 0.9772, 0.9927] 
+2024-09-11 07:43:35.672120: Epoch time: 266.25 s 
+2024-09-11 07:43:35.672956: Yayy! New best EMA pseudo Dice: 0.961 
+2024-09-11 07:43:41.194665:  
+2024-09-11 07:43:41.195712: Epoch 912 
+2024-09-11 07:43:41.196752: Current learning rate: 0.00112 
+2024-09-11 07:47:52.456986: train_loss -0.7523 
+2024-09-11 07:47:52.459291: val_loss -0.7337 
+2024-09-11 07:47:52.462662: Pseudo dice [0.9654, 0.9614, 0.9776, 0.9534, 0.8971, 0.9761, 0.9914] 
+2024-09-11 07:47:52.464623: Epoch time: 251.27 s 
+2024-09-11 07:47:55.451865:  
+2024-09-11 07:47:55.464856: Epoch 913 
+2024-09-11 07:47:55.465919: Current learning rate: 0.00111 
+2024-09-11 07:52:10.987466: train_loss -0.7295 
+2024-09-11 07:52:11.022217: val_loss -0.7964 
+2024-09-11 07:52:11.023154: Pseudo dice [0.9679, 0.9615, 0.9813, 0.9571, 0.9005, 0.9769, 0.9922] 
+2024-09-11 07:52:11.023968: Epoch time: 255.54 s 
+2024-09-11 07:52:11.024862: Yayy! New best EMA pseudo Dice: 0.9611 
+2024-09-11 07:52:19.665240:  
+2024-09-11 07:52:19.666380: Epoch 914 
+2024-09-11 07:52:19.667264: Current learning rate: 0.0011 
+2024-09-11 07:56:28.464936: train_loss -0.7566 
+2024-09-11 07:56:28.467633: val_loss -0.7697 
+2024-09-11 07:56:28.469032: Pseudo dice [0.9675, 0.9596, 0.9814, 0.9606, 0.8913, 0.9746, 0.9921] 
+2024-09-11 07:56:28.469991: Epoch time: 248.8 s 
+2024-09-11 07:56:30.963891:  
+2024-09-11 07:56:30.965012: Epoch 915 
+2024-09-11 07:56:30.965822: Current learning rate: 0.00109 
+2024-09-11 08:00:52.992411: train_loss -0.7508 
+2024-09-11 08:00:52.994988: val_loss -0.7453 
+2024-09-11 08:00:52.996210: Pseudo dice [0.9696, 0.9615, 0.9829, 0.956, 0.9003, 0.9784, 0.9938] 
+2024-09-11 08:00:52.997212: Epoch time: 262.03 s 
+2024-09-11 08:00:52.998293: Yayy! New best EMA pseudo Dice: 0.9613 
+2024-09-11 08:00:59.282679:  
+2024-09-11 08:00:59.283898: Epoch 916 
+2024-09-11 08:00:59.284843: Current learning rate: 0.00108 
+2024-09-11 08:05:23.211220: train_loss -0.7533 
+2024-09-11 08:05:23.226760: val_loss -0.761 
+2024-09-11 08:05:23.228145: Pseudo dice [0.9661, 0.9612, 0.9809, 0.9495, 0.888, 0.9744, 0.9899] 
+2024-09-11 08:05:23.229370: Epoch time: 263.93 s 
+2024-09-11 08:05:26.383017:  
+2024-09-11 08:05:26.384238: Epoch 917 
+2024-09-11 08:05:26.385168: Current learning rate: 0.00106 
+2024-09-11 08:09:53.407262: train_loss -0.7573 
+2024-09-11 08:09:53.422227: val_loss -0.7769 
+2024-09-11 08:09:53.423178: Pseudo dice [0.9676, 0.9619, 0.9795, 0.9589, 0.8977, 0.9767, 0.992] 
+2024-09-11 08:09:53.424092: Epoch time: 267.03 s 
+2024-09-11 08:09:55.838941:  
+2024-09-11 08:09:55.852922: Epoch 918 
+2024-09-11 08:09:55.854269: Current learning rate: 0.00105 
+2024-09-11 08:14:27.264977: train_loss -0.7671 
+2024-09-11 08:14:27.280342: val_loss -0.7428 
+2024-09-11 08:14:27.281346: Pseudo dice [0.9696, 0.961, 0.9822, 0.9534, 0.9013, 0.9788, 0.9934] 
+2024-09-11 08:14:27.282194: Epoch time: 271.43 s 
+2024-09-11 08:14:27.282973: Yayy! New best EMA pseudo Dice: 0.9613 
+2024-09-11 08:14:32.622827:  
+2024-09-11 08:14:32.623771: Epoch 919 
+2024-09-11 08:14:32.624683: Current learning rate: 0.00104 
+2024-09-11 08:18:52.694377: train_loss -0.7656 
+2024-09-11 08:18:52.696897: val_loss -0.7876 
+2024-09-11 08:18:52.697920: Pseudo dice [0.9677, 0.9599, 0.9828, 0.9553, 0.8968, 0.9746, 0.9921] 
+2024-09-11 08:18:52.698739: Epoch time: 260.07 s 
+2024-09-11 08:18:52.699510: Yayy! New best EMA pseudo Dice: 0.9613 
+2024-09-11 08:18:58.868754:  
+2024-09-11 08:18:58.869789: Epoch 920 
+2024-09-11 08:18:58.870702: Current learning rate: 0.00103 
+2024-09-11 08:23:06.593619: train_loss -0.7454 
+2024-09-11 08:23:06.612321: val_loss -0.7629 
+2024-09-11 08:23:06.613984: Pseudo dice [0.9667, 0.9609, 0.9799, 0.9476, 0.8772, 0.9738, 0.9906] 
+2024-09-11 08:23:06.615607: Epoch time: 247.73 s 
+2024-09-11 08:23:09.148295:  
+2024-09-11 08:23:09.149673: Epoch 921 
+2024-09-11 08:23:09.150871: Current learning rate: 0.00102 
+2024-09-11 08:27:15.171075: train_loss -0.7559 
+2024-09-11 08:27:15.186013: val_loss -0.7543 
+2024-09-11 08:27:15.187144: Pseudo dice [0.9683, 0.9626, 0.9793, 0.9519, 0.886, 0.9763, 0.9922] 
+2024-09-11 08:27:15.188299: Epoch time: 246.03 s 
+2024-09-11 08:27:18.770245:  
+2024-09-11 08:27:18.771507: Epoch 922 
+2024-09-11 08:27:18.772423: Current learning rate: 0.00101 
+2024-09-11 08:31:42.832949: train_loss -0.7473 
+2024-09-11 08:31:42.848210: val_loss -0.7905 
+2024-09-11 08:31:42.849806: Pseudo dice [0.9647, 0.9607, 0.9798, 0.9579, 0.8872, 0.9726, 0.9897] 
+2024-09-11 08:31:42.851027: Epoch time: 264.07 s 
+2024-09-11 08:31:45.811810:  
+2024-09-11 08:31:45.812869: Epoch 923 
+2024-09-11 08:31:45.813728: Current learning rate: 0.001 
+2024-09-11 08:35:43.546498: train_loss -0.7543 
+2024-09-11 08:35:43.574221: val_loss -0.741 
+2024-09-11 08:35:43.575313: Pseudo dice [0.9679, 0.9605, 0.9805, 0.9541, 0.9013, 0.9792, 0.9932] 
+2024-09-11 08:35:43.576406: Epoch time: 237.74 s 
+2024-09-11 08:35:46.958209:  
+2024-09-11 08:35:46.959536: Epoch 924 
+2024-09-11 08:35:46.960535: Current learning rate: 0.00098 
+2024-09-11 08:40:18.003703: train_loss -0.7576 
+2024-09-11 08:40:18.018072: val_loss -0.7457 
+2024-09-11 08:40:18.018886: Pseudo dice [0.9679, 0.9588, 0.98, 0.9515, 0.8924, 0.9736, 0.9917] 
+2024-09-11 08:40:18.019685: Epoch time: 271.05 s 
+2024-09-11 08:40:20.383985:  
+2024-09-11 08:40:20.392510: Epoch 925 
+2024-09-11 08:40:20.393448: Current learning rate: 0.00097 
+2024-09-11 08:44:50.291665: train_loss -0.7606 
+2024-09-11 08:44:50.294539: val_loss -0.7427 
+2024-09-11 08:44:50.295760: Pseudo dice [0.9705, 0.9645, 0.9814, 0.9576, 0.9026, 0.9785, 0.9914] 
+2024-09-11 08:44:50.297002: Epoch time: 269.91 s 
+2024-09-11 08:44:52.704976:  
+2024-09-11 08:44:52.706521: Epoch 926 
+2024-09-11 08:44:52.707743: Current learning rate: 0.00096 
+2024-09-11 08:49:02.557230: train_loss -0.7623 
+2024-09-11 08:49:02.559686: val_loss -0.7516 
+2024-09-11 08:49:02.560687: Pseudo dice [0.9679, 0.9634, 0.9807, 0.9569, 0.8941, 0.9766, 0.9924] 
+2024-09-11 08:49:02.561591: Epoch time: 249.86 s 
+2024-09-11 08:49:05.071858:  
+2024-09-11 08:49:05.073080: Epoch 927 
+2024-09-11 08:49:05.074116: Current learning rate: 0.00095 
+2024-09-11 08:53:31.511159: train_loss -0.7777 
+2024-09-11 08:53:31.513991: val_loss -0.7461 
+2024-09-11 08:53:31.515135: Pseudo dice [0.9683, 0.9608, 0.9827, 0.9541, 0.8953, 0.9759, 0.9909] 
+2024-09-11 08:53:31.516167: Epoch time: 266.44 s 
+2024-09-11 08:53:35.237016:  
+2024-09-11 08:53:35.238127: Epoch 928 
+2024-09-11 08:53:35.239002: Current learning rate: 0.00094 
+2024-09-11 08:57:46.749549: train_loss -0.7525 
+2024-09-11 08:57:46.763717: val_loss -0.7588 
+2024-09-11 08:57:46.765603: Pseudo dice [0.968, 0.9604, 0.9817, 0.9538, 0.9003, 0.9778, 0.9918] 
+2024-09-11 08:57:46.767316: Epoch time: 251.52 s 
+2024-09-11 08:57:49.904560:  
+2024-09-11 08:57:49.905654: Epoch 929 
+2024-09-11 08:57:49.906587: Current learning rate: 0.00092 
+2024-09-11 09:02:04.392052: train_loss -0.7711 
+2024-09-11 09:02:04.408714: val_loss -0.7308 
+2024-09-11 09:02:04.410290: Pseudo dice [0.9696, 0.9603, 0.9827, 0.9599, 0.8997, 0.9784, 0.9931] 
+2024-09-11 09:02:04.411726: Epoch time: 254.49 s 
+2024-09-11 09:02:04.413197: Yayy! New best EMA pseudo Dice: 0.9613 
+2024-09-11 09:02:10.218758:  
+2024-09-11 09:02:10.219818: Epoch 930 
+2024-09-11 09:02:10.220773: Current learning rate: 0.00091 
+2024-09-11 09:06:41.615867: train_loss -0.7664 
+2024-09-11 09:06:41.634505: val_loss -0.7954 
+2024-09-11 09:06:41.635555: Pseudo dice [0.9683, 0.9599, 0.9815, 0.9555, 0.8859, 0.9771, 0.9907] 
+2024-09-11 09:06:41.636966: Epoch time: 271.4 s 
+2024-09-11 09:06:44.576936:  
+2024-09-11 09:06:44.578174: Epoch 931 
+2024-09-11 09:06:44.579063: Current learning rate: 0.0009 
+2024-09-11 09:10:53.011118: train_loss -0.7403 
+2024-09-11 09:10:53.022735: val_loss -0.788 
+2024-09-11 09:10:53.024102: Pseudo dice [0.9706, 0.9611, 0.9838, 0.9565, 0.8959, 0.9761, 0.9934] 
+2024-09-11 09:10:53.025874: Epoch time: 248.44 s 
+2024-09-11 09:10:55.258239:  
+2024-09-11 09:10:55.259472: Epoch 932 
+2024-09-11 09:10:55.260378: Current learning rate: 0.00089 
+2024-09-11 09:15:02.170901: train_loss -0.7595 
+2024-09-11 09:15:02.186184: val_loss -0.7308 
+2024-09-11 09:15:02.187125: Pseudo dice [0.9677, 0.9627, 0.9816, 0.9578, 0.8919, 0.9774, 0.9929] 
+2024-09-11 09:15:02.188067: Epoch time: 246.92 s 
+2024-09-11 09:15:02.188971: Yayy! New best EMA pseudo Dice: 0.9614 
+2024-09-11 09:15:08.598662:  
+2024-09-11 09:15:08.599732: Epoch 933 
+2024-09-11 09:15:08.600814: Current learning rate: 0.00088 
+2024-09-11 09:19:13.780659: train_loss -0.7608 
+2024-09-11 09:19:13.806079: val_loss -0.7253 
+2024-09-11 09:19:13.807025: Pseudo dice [0.9697, 0.9624, 0.9812, 0.9522, 0.8937, 0.974, 0.9926] 
+2024-09-11 09:19:13.807930: Epoch time: 245.19 s 
+2024-09-11 09:19:16.918927:  
+2024-09-11 09:19:16.932929: Epoch 934 
+2024-09-11 09:19:16.934030: Current learning rate: 0.00087 
+2024-09-11 09:23:40.655202: train_loss -0.7616 
+2024-09-11 09:23:40.669581: val_loss -0.7899 
+2024-09-11 09:23:40.670483: Pseudo dice [0.9671, 0.9603, 0.9797, 0.9508, 0.8859, 0.9738, 0.9905] 
+2024-09-11 09:23:40.671397: Epoch time: 263.74 s 
+2024-09-11 09:23:43.209140:  
+2024-09-11 09:23:43.210144: Epoch 935 
+2024-09-11 09:23:43.210898: Current learning rate: 0.00085 
+2024-09-11 09:27:57.010201: train_loss -0.745 
+2024-09-11 09:27:57.038164: val_loss -0.7714 
+2024-09-11 09:27:57.039228: Pseudo dice [0.965, 0.9579, 0.9778, 0.9473, 0.8913, 0.9718, 0.9918] 
+2024-09-11 09:27:57.040359: Epoch time: 253.81 s 
+2024-09-11 09:27:59.809429:  
+2024-09-11 09:27:59.810646: Epoch 936 
+2024-09-11 09:27:59.812175: Current learning rate: 0.00084 
+2024-09-11 09:32:22.859803: train_loss -0.7568 
+2024-09-11 09:32:22.873101: val_loss -0.7543 
+2024-09-11 09:32:22.874133: Pseudo dice [0.9696, 0.9621, 0.9834, 0.956, 0.8931, 0.9764, 0.992] 
+2024-09-11 09:32:22.875098: Epoch time: 263.05 s 
+2024-09-11 09:32:25.457957:  
+2024-09-11 09:32:25.459091: Epoch 937 
+2024-09-11 09:32:25.459965: Current learning rate: 0.00083 
+2024-09-11 09:36:56.823257: train_loss -0.761 
+2024-09-11 09:36:56.826056: val_loss -0.7703 
+2024-09-11 09:36:56.827053: Pseudo dice [0.9682, 0.9614, 0.9803, 0.9592, 0.8873, 0.9758, 0.9927] 
+2024-09-11 09:36:56.827936: Epoch time: 271.37 s 
+2024-09-11 09:36:59.457748:  
+2024-09-11 09:36:59.458987: Epoch 938 
+2024-09-11 09:36:59.459864: Current learning rate: 0.00082 
+2024-09-11 09:41:05.878016: train_loss -0.7716 
+2024-09-11 09:41:05.896201: val_loss -0.7612 
+2024-09-11 09:41:05.897270: Pseudo dice [0.9683, 0.9605, 0.9805, 0.957, 0.8959, 0.9771, 0.9925] 
+2024-09-11 09:41:05.898156: Epoch time: 246.43 s 
+2024-09-11 09:41:10.204725:  
+2024-09-11 09:41:10.205964: Epoch 939 
+2024-09-11 09:41:10.206994: Current learning rate: 0.00081 
+2024-09-11 09:45:15.820887: train_loss -0.7734 
+2024-09-11 09:45:15.823267: val_loss -0.752 
+2024-09-11 09:45:15.824153: Pseudo dice [0.9674, 0.9622, 0.979, 0.9588, 0.9004, 0.977, 0.9922] 
+2024-09-11 09:45:15.825151: Epoch time: 245.62 s 
+2024-09-11 09:45:18.541648:  
+2024-09-11 09:45:18.543062: Epoch 940 
+2024-09-11 09:45:18.544200: Current learning rate: 0.00079 
+2024-09-11 09:49:37.468518: train_loss -0.7482 
+2024-09-11 09:49:37.471064: val_loss -0.7515 
+2024-09-11 09:49:37.472030: Pseudo dice [0.9692, 0.9609, 0.9832, 0.9521, 0.8983, 0.9755, 0.9922] 
+2024-09-11 09:49:37.473090: Epoch time: 258.93 s 
+2024-09-11 09:49:39.948511:  
+2024-09-11 09:49:39.949609: Epoch 941 
+2024-09-11 09:49:39.950590: Current learning rate: 0.00078 
+2024-09-11 09:54:02.748780: train_loss -0.763 
+2024-09-11 09:54:02.762614: val_loss -0.7756 
+2024-09-11 09:54:02.763978: Pseudo dice [0.9685, 0.9597, 0.9812, 0.95, 0.8944, 0.9752, 0.9924] 
+2024-09-11 09:54:02.765341: Epoch time: 262.8 s 
+2024-09-11 09:54:05.568476:  
+2024-09-11 09:54:05.569722: Epoch 942 
+2024-09-11 09:54:05.570613: Current learning rate: 0.00077 
+2024-09-11 09:58:29.661246: train_loss -0.7661 
+2024-09-11 09:58:29.663887: val_loss -0.7626 
+2024-09-11 09:58:29.664824: Pseudo dice [0.9689, 0.9619, 0.9836, 0.9612, 0.8866, 0.9733, 0.9906] 
+2024-09-11 09:58:29.665709: Epoch time: 264.1 s 
+2024-09-11 09:58:31.762126:  
+2024-09-11 09:58:31.763774: Epoch 943 
+2024-09-11 09:58:31.764993: Current learning rate: 0.00076 
+2024-09-11 10:02:42.237260: train_loss -0.7659 
+2024-09-11 10:02:42.250286: val_loss -0.791 
+2024-09-11 10:02:42.251252: Pseudo dice [0.9671, 0.9589, 0.9831, 0.9568, 0.8943, 0.977, 0.9923] 
+2024-09-11 10:02:42.252145: Epoch time: 250.48 s 
+2024-09-11 10:02:45.448495:  
+2024-09-11 10:02:45.449687: Epoch 944 
+2024-09-11 10:02:45.450596: Current learning rate: 0.00075 
+2024-09-11 10:07:04.149459: train_loss -0.767 
+2024-09-11 10:07:04.151973: val_loss -0.7633 
+2024-09-11 10:07:04.153176: Pseudo dice [0.9683, 0.9621, 0.9809, 0.9585, 0.9022, 0.9784, 0.9929] 
+2024-09-11 10:07:04.154303: Epoch time: 258.71 s 
+2024-09-11 10:07:06.156991:  
+2024-09-11 10:07:06.158021: Epoch 945 
+2024-09-11 10:07:06.158975: Current learning rate: 0.00074 
+2024-09-11 10:11:03.632760: train_loss -0.7566 
+2024-09-11 10:11:03.650357: val_loss -0.7792 
+2024-09-11 10:11:03.651270: Pseudo dice [0.9697, 0.9634, 0.9829, 0.9549, 0.9027, 0.9758, 0.9929] 
+2024-09-11 10:11:03.652110: Epoch time: 237.48 s 
+2024-09-11 10:11:03.653034: Yayy! New best EMA pseudo Dice: 0.9614 
+2024-09-11 10:11:10.639747:  
+2024-09-11 10:11:10.640836: Epoch 946 
+2024-09-11 10:11:10.641740: Current learning rate: 0.00072 
+2024-09-11 10:15:28.520560: train_loss -0.7756 
+2024-09-11 10:15:28.523004: val_loss -0.7699 
+2024-09-11 10:15:28.523918: Pseudo dice [0.9686, 0.9617, 0.9823, 0.9558, 0.8983, 0.9776, 0.9935] 
+2024-09-11 10:15:28.524806: Epoch time: 257.88 s 
+2024-09-11 10:15:28.525658: Yayy! New best EMA pseudo Dice: 0.9616 
+2024-09-11 10:15:33.938874:  
+2024-09-11 10:15:33.939874: Epoch 947 
+2024-09-11 10:15:33.940738: Current learning rate: 0.00071 
+2024-09-11 10:19:44.571703: train_loss -0.7781 
+2024-09-11 10:19:44.598437: val_loss -0.7604 
+2024-09-11 10:19:44.600027: Pseudo dice [0.9653, 0.9626, 0.9793, 0.9575, 0.905, 0.9742, 0.9932] 
+2024-09-11 10:19:44.601341: Epoch time: 250.64 s 
+2024-09-11 10:19:44.602401: Yayy! New best EMA pseudo Dice: 0.9616 
+2024-09-11 10:19:49.703750:  
+2024-09-11 10:19:49.704767: Epoch 948 
+2024-09-11 10:19:49.705577: Current learning rate: 0.0007 
+2024-09-11 10:23:38.538011: train_loss -0.768 
+2024-09-11 10:23:38.550249: val_loss -0.7296 
+2024-09-11 10:23:38.551395: Pseudo dice [0.9672, 0.963, 0.9804, 0.9597, 0.8961, 0.9708, 0.9915] 
+2024-09-11 10:23:38.552420: Epoch time: 228.84 s 
+2024-09-11 10:23:42.699131:  
+2024-09-11 10:23:42.708749: Epoch 949 
+2024-09-11 10:23:42.709824: Current learning rate: 0.00069 
+2024-09-11 10:28:06.733817: train_loss -0.7712 
+2024-09-11 10:28:06.736451: val_loss -0.7403 
+2024-09-11 10:28:06.737439: Pseudo dice [0.9641, 0.9615, 0.9762, 0.9514, 0.898, 0.971, 0.992] 
+2024-09-11 10:28:06.738263: Epoch time: 264.04 s 
+2024-09-11 10:28:13.168733:  
+2024-09-11 10:28:13.169751: Epoch 950 
+2024-09-11 10:28:13.170622: Current learning rate: 0.00067 
+2024-09-11 10:32:24.011608: train_loss -0.7677 
+2024-09-11 10:32:24.014278: val_loss -0.7573 
+2024-09-11 10:32:24.015496: Pseudo dice [0.9677, 0.9615, 0.9814, 0.9594, 0.9, 0.9752, 0.9924] 
+2024-09-11 10:32:24.016535: Epoch time: 250.85 s 
+2024-09-11 10:32:27.223788:  
+2024-09-11 10:32:27.224876: Epoch 951 
+2024-09-11 10:32:27.225770: Current learning rate: 0.00066 
+2024-09-11 10:37:08.296685: train_loss -0.7505 
+2024-09-11 10:37:08.299106: val_loss -0.7845 
+2024-09-11 10:37:08.300176: Pseudo dice [0.9704, 0.9619, 0.984, 0.9609, 0.8984, 0.9773, 0.9924] 
+2024-09-11 10:37:08.301032: Epoch time: 281.08 s 
+2024-09-11 10:37:08.301878: Yayy! New best EMA pseudo Dice: 0.9617 
+2024-09-11 10:37:13.676119:  
+2024-09-11 10:37:13.677203: Epoch 952 
+2024-09-11 10:37:13.678171: Current learning rate: 0.00065 
+2024-09-11 10:41:30.900789: train_loss -0.7687 
+2024-09-11 10:41:30.904145: val_loss -0.7536 
+2024-09-11 10:41:30.905716: Pseudo dice [0.9692, 0.9647, 0.9826, 0.9613, 0.9021, 0.9754, 0.9925] 
+2024-09-11 10:41:30.907202: Epoch time: 257.23 s 
+2024-09-11 10:41:30.909037: Yayy! New best EMA pseudo Dice: 0.9619 
+2024-09-11 10:41:35.637291:  
+2024-09-11 10:41:35.638612: Epoch 953 
+2024-09-11 10:41:35.639542: Current learning rate: 0.00064 
+2024-09-11 10:45:45.168978: train_loss -0.7444 
+2024-09-11 10:45:45.171564: val_loss -0.7688 
+2024-09-11 10:45:45.172622: Pseudo dice [0.9703, 0.9625, 0.9819, 0.9546, 0.8953, 0.9782, 0.9923] 
+2024-09-11 10:45:45.173801: Epoch time: 249.53 s 
+2024-09-11 10:45:45.174731: Yayy! New best EMA pseudo Dice: 0.9619 
+2024-09-11 10:45:50.813589:  
+2024-09-11 10:45:50.814742: Epoch 954 
+2024-09-11 10:45:50.815639: Current learning rate: 0.00063 
+2024-09-11 10:50:24.180819: train_loss -0.7785 
+2024-09-11 10:50:24.194451: val_loss -0.774 
+2024-09-11 10:50:24.195322: Pseudo dice [0.9688, 0.9627, 0.9807, 0.959, 0.8993, 0.9773, 0.9929] 
+2024-09-11 10:50:24.196207: Epoch time: 273.37 s 
+2024-09-11 10:50:24.197034: Yayy! New best EMA pseudo Dice: 0.962 
+2024-09-11 10:50:29.907140:  
+2024-09-11 10:50:29.908403: Epoch 955 
+2024-09-11 10:50:29.909360: Current learning rate: 0.00061 
+2024-09-11 10:54:36.875162: train_loss -0.7453 
+2024-09-11 10:54:36.890401: val_loss -0.7905 
+2024-09-11 10:54:36.891334: Pseudo dice [0.9711, 0.9623, 0.9831, 0.9551, 0.9009, 0.9793, 0.9926] 
+2024-09-11 10:54:36.892220: Epoch time: 246.97 s 
+2024-09-11 10:54:36.893054: Yayy! New best EMA pseudo Dice: 0.9622 
+2024-09-11 10:54:43.173856:  
+2024-09-11 10:54:43.174947: Epoch 956 
+2024-09-11 10:54:43.175773: Current learning rate: 0.0006 
+2024-09-11 10:59:14.641101: train_loss -0.7717 
+2024-09-11 10:59:14.643530: val_loss -0.7897 
+2024-09-11 10:59:14.660195: Pseudo dice [0.9716, 0.9641, 0.9851, 0.9584, 0.898, 0.9764, 0.9919] 
+2024-09-11 10:59:14.661227: Epoch time: 271.47 s 
+2024-09-11 10:59:14.662130: Yayy! New best EMA pseudo Dice: 0.9623 
+2024-09-11 10:59:19.848502:  
+2024-09-11 10:59:19.849865: Epoch 957 
+2024-09-11 10:59:19.850779: Current learning rate: 0.00059 
+2024-09-11 11:03:30.148680: train_loss -0.7567 
+2024-09-11 11:03:30.166423: val_loss -0.7883 
+2024-09-11 11:03:30.167554: Pseudo dice [0.9701, 0.9645, 0.9809, 0.9568, 0.9057, 0.9787, 0.9941] 
+2024-09-11 11:03:30.168635: Epoch time: 250.3 s 
+2024-09-11 11:03:30.169647: Yayy! New best EMA pseudo Dice: 0.9625 
+2024-09-11 11:03:36.541507:  
+2024-09-11 11:03:36.542454: Epoch 958 
+2024-09-11 11:03:36.543231: Current learning rate: 0.00058 
+2024-09-11 11:07:59.373691: train_loss -0.7728 
+2024-09-11 11:07:59.389529: val_loss -0.7739 
+2024-09-11 11:07:59.390650: Pseudo dice [0.9658, 0.9606, 0.9786, 0.9522, 0.8853, 0.975, 0.9906] 
+2024-09-11 11:07:59.391612: Epoch time: 262.84 s 
+2024-09-11 11:08:01.437805:  
+2024-09-11 11:08:01.439093: Epoch 959 
+2024-09-11 11:08:01.440165: Current learning rate: 0.00056 
+2024-09-11 11:12:21.098939: train_loss -0.7501 
+2024-09-11 11:12:21.115607: val_loss -0.8076 
+2024-09-11 11:12:21.116545: Pseudo dice [0.9694, 0.9615, 0.9826, 0.9575, 0.9026, 0.978, 0.9922] 
+2024-09-11 11:12:21.117344: Epoch time: 259.67 s 
+2024-09-11 11:12:23.858516:  
+2024-09-11 11:12:23.859805: Epoch 960 
+2024-09-11 11:12:23.860932: Current learning rate: 0.00055 
+2024-09-11 11:16:56.551478: train_loss -0.758 
+2024-09-11 11:16:56.566367: val_loss -0.7572 
+2024-09-11 11:16:56.567739: Pseudo dice [0.9701, 0.9627, 0.9839, 0.9501, 0.8937, 0.9758, 0.9919] 
+2024-09-11 11:16:56.569148: Epoch time: 272.7 s 
+2024-09-11 11:17:00.123444:  
+2024-09-11 11:17:00.124679: Epoch 961 
+2024-09-11 11:17:00.125571: Current learning rate: 0.00054 
+2024-09-11 11:21:19.729711: train_loss -0.757 
+2024-09-11 11:21:19.731974: val_loss -0.7884 
+2024-09-11 11:21:19.732912: Pseudo dice [0.9718, 0.9641, 0.9841, 0.9567, 0.9002, 0.9796, 0.9932] 
+2024-09-11 11:21:19.733728: Epoch time: 259.61 s 
+2024-09-11 11:21:22.890033:  
+2024-09-11 11:21:22.891289: Epoch 962 
+2024-09-11 11:21:22.892284: Current learning rate: 0.00053 
+2024-09-11 11:25:43.793201: train_loss -0.775 
+2024-09-11 11:25:43.795660: val_loss -0.7716 
+2024-09-11 11:25:43.796688: Pseudo dice [0.9688, 0.9599, 0.9842, 0.9554, 0.9067, 0.979, 0.9938] 
+2024-09-11 11:25:43.797681: Epoch time: 260.91 s 
+2024-09-11 11:25:46.184753:  
+2024-09-11 11:25:46.186539: Epoch 963 
+2024-09-11 11:25:46.187673: Current learning rate: 0.00051 
+2024-09-11 11:30:00.879099: train_loss -0.7647 
+2024-09-11 11:30:00.881492: val_loss -0.7884 
+2024-09-11 11:30:00.882409: Pseudo dice [0.9701, 0.9625, 0.983, 0.9577, 0.8965, 0.975, 0.9922] 
+2024-09-11 11:30:00.883263: Epoch time: 254.7 s 
+2024-09-11 11:30:03.768404:  
+2024-09-11 11:30:03.769610: Epoch 964 
+2024-09-11 11:30:03.770519: Current learning rate: 0.0005 
+2024-09-11 11:34:19.642773: train_loss -0.7772 
+2024-09-11 11:34:19.645524: val_loss -0.7773 
+2024-09-11 11:34:19.647798: Pseudo dice [0.969, 0.9616, 0.9803, 0.9558, 0.9027, 0.9772, 0.9926] 
+2024-09-11 11:34:19.648657: Epoch time: 255.88 s 
+2024-09-11 11:34:22.522580:  
+2024-09-11 11:34:22.536093: Epoch 965 
+2024-09-11 11:34:22.537188: Current learning rate: 0.00049 
+2024-09-11 11:38:34.908293: train_loss -0.7597 
+2024-09-11 11:38:34.910764: val_loss -0.7398 
+2024-09-11 11:38:34.911610: Pseudo dice [0.9665, 0.9621, 0.9811, 0.9496, 0.8938, 0.9722, 0.9902] 
+2024-09-11 11:38:34.917102: Epoch time: 252.39 s 
+2024-09-11 11:38:37.672382:  
+2024-09-11 11:38:37.673411: Epoch 966 
+2024-09-11 11:38:37.674285: Current learning rate: 0.00048 
+2024-09-11 11:42:54.640455: train_loss -0.746 
+2024-09-11 11:42:54.642926: val_loss -0.7642 
+2024-09-11 11:42:54.662355: Pseudo dice [0.9696, 0.9616, 0.9803, 0.9597, 0.8945, 0.9779, 0.9911] 
+2024-09-11 11:42:54.672341: Epoch time: 256.97 s 
+2024-09-11 11:42:57.128911:  
+2024-09-11 11:42:57.130191: Epoch 967 
+2024-09-11 11:42:57.131051: Current learning rate: 0.00046 
+2024-09-11 11:47:04.002091: train_loss -0.7738 
+2024-09-11 11:47:04.018485: val_loss -0.752 
+2024-09-11 11:47:04.019714: Pseudo dice [0.97, 0.9605, 0.9839, 0.957, 0.8951, 0.978, 0.9922] 
+2024-09-11 11:47:04.020711: Epoch time: 246.88 s 
+2024-09-11 11:47:06.577738:  
+2024-09-11 11:47:06.578834: Epoch 968 
+2024-09-11 11:47:06.579642: Current learning rate: 0.00045 
+2024-09-11 11:51:25.322892: train_loss -0.7827 
+2024-09-11 11:51:25.325197: val_loss -0.792 
+2024-09-11 11:51:25.326036: Pseudo dice [0.9694, 0.9608, 0.983, 0.9571, 0.8954, 0.9756, 0.9926] 
+2024-09-11 11:51:25.326827: Epoch time: 258.75 s 
+2024-09-11 11:51:27.947942:  
+2024-09-11 11:51:27.949189: Epoch 969 
+2024-09-11 11:51:27.950036: Current learning rate: 0.00044 
+2024-09-11 11:55:42.773873: train_loss -0.7582 
+2024-09-11 11:55:42.792164: val_loss -0.7374 
+2024-09-11 11:55:42.793238: Pseudo dice [0.9643, 0.9618, 0.9761, 0.9524, 0.8985, 0.9753, 0.9911] 
+2024-09-11 11:55:42.794133: Epoch time: 254.83 s 
+2024-09-11 11:55:44.854235:  
+2024-09-11 11:55:44.865260: Epoch 970 
+2024-09-11 11:55:44.866244: Current learning rate: 0.00043 
+2024-09-11 11:59:57.450652: train_loss -0.7726 
+2024-09-11 11:59:57.454373: val_loss -0.7456 
+2024-09-11 11:59:57.456535: Pseudo dice [0.9679, 0.9601, 0.9791, 0.9535, 0.8981, 0.9776, 0.993] 
+2024-09-11 11:59:57.458340: Epoch time: 252.6 s 
+2024-09-11 12:00:00.760573:  
+2024-09-11 12:00:00.761593: Epoch 971 
+2024-09-11 12:00:00.762515: Current learning rate: 0.00041 
+2024-09-11 12:04:20.464401: train_loss -0.7714 
+2024-09-11 12:04:20.482563: val_loss -0.7624 
+2024-09-11 12:04:20.483717: Pseudo dice [0.9714, 0.962, 0.9847, 0.9582, 0.9001, 0.9791, 0.9933] 
+2024-09-11 12:04:20.485023: Epoch time: 259.71 s 
+2024-09-11 12:04:23.909567:  
+2024-09-11 12:04:23.927719: Epoch 972 
+2024-09-11 12:04:23.936457: Current learning rate: 0.0004 
+2024-09-11 12:08:34.829508: train_loss -0.755 
+2024-09-11 12:08:34.846513: val_loss -0.7837 
+2024-09-11 12:08:34.847462: Pseudo dice [0.9681, 0.9621, 0.9805, 0.9581, 0.9041, 0.9788, 0.9922] 
+2024-09-11 12:08:34.848390: Epoch time: 250.93 s 
+2024-09-11 12:08:37.602744:  
+2024-09-11 12:08:37.603952: Epoch 973 
+2024-09-11 12:08:37.605196: Current learning rate: 0.00039 
+2024-09-11 12:12:44.745166: train_loss -0.7658 
+2024-09-11 12:12:44.759796: val_loss -0.7664 
+2024-09-11 12:12:44.760762: Pseudo dice [0.9701, 0.9622, 0.9826, 0.9546, 0.9015, 0.9778, 0.993] 
+2024-09-11 12:12:44.761606: Epoch time: 247.15 s 
+2024-09-11 12:12:48.509961:  
+2024-09-11 12:12:48.511137: Epoch 974 
+2024-09-11 12:12:48.511969: Current learning rate: 0.00037 
+2024-09-11 12:17:00.820264: train_loss -0.7561 
+2024-09-11 12:17:00.838116: val_loss -0.7661 
+2024-09-11 12:17:00.839053: Pseudo dice [0.9679, 0.962, 0.9811, 0.9491, 0.8979, 0.9767, 0.9929] 
+2024-09-11 12:17:00.839855: Epoch time: 252.31 s 
+2024-09-11 12:17:03.800106:  
+2024-09-11 12:17:03.801229: Epoch 975 
+2024-09-11 12:17:03.802115: Current learning rate: 0.00036 
+2024-09-11 12:21:18.362956: train_loss -0.7663 
+2024-09-11 12:21:18.365547: val_loss -0.739 
+2024-09-11 12:21:18.366482: Pseudo dice [0.9701, 0.9634, 0.9826, 0.9567, 0.8929, 0.9775, 0.9914] 
+2024-09-11 12:21:18.368720: Epoch time: 254.57 s 
+2024-09-11 12:21:20.657519:  
+2024-09-11 12:21:20.658730: Epoch 976 
+2024-09-11 12:21:20.659669: Current learning rate: 0.00035 
+2024-09-11 12:25:36.345321: train_loss -0.7738 
+2024-09-11 12:25:36.358181: val_loss -0.7832 
+2024-09-11 12:25:36.359024: Pseudo dice [0.9667, 0.9587, 0.9807, 0.9481, 0.8798, 0.9733, 0.9907] 
+2024-09-11 12:25:36.359831: Epoch time: 255.69 s 
+2024-09-11 12:25:40.197331:  
+2024-09-11 12:25:40.206364: Epoch 977 
+2024-09-11 12:25:40.207462: Current learning rate: 0.00034 
+2024-09-11 12:29:53.495682: train_loss -0.7581 
+2024-09-11 12:29:53.510183: val_loss -0.7497 
+2024-09-11 12:29:53.511133: Pseudo dice [0.971, 0.9636, 0.9832, 0.9563, 0.8935, 0.9776, 0.9922] 
+2024-09-11 12:29:53.511915: Epoch time: 253.3 s 
+2024-09-11 12:29:57.375715:  
+2024-09-11 12:29:57.377044: Epoch 978 
+2024-09-11 12:29:57.377893: Current learning rate: 0.00032 
+2024-09-11 12:34:01.702424: train_loss -0.7654 
+2024-09-11 12:34:01.727551: val_loss -0.7919 
+2024-09-11 12:34:01.729389: Pseudo dice [0.9694, 0.9638, 0.9828, 0.9571, 0.9052, 0.9777, 0.9931] 
+2024-09-11 12:34:01.731446: Epoch time: 244.33 s 
+2024-09-11 12:34:04.956527:  
+2024-09-11 12:34:04.971535: Epoch 979 
+2024-09-11 12:34:04.972583: Current learning rate: 0.00031 
+2024-09-11 12:38:37.783706: train_loss -0.7619 
+2024-09-11 12:38:37.796341: val_loss -0.732 
+2024-09-11 12:38:37.797319: Pseudo dice [0.9715, 0.9635, 0.9853, 0.9608, 0.9045, 0.9771, 0.9941] 
+2024-09-11 12:38:37.798322: Epoch time: 272.83 s 
+2024-09-11 12:38:41.011697:  
+2024-09-11 12:38:41.012821: Epoch 980 
+2024-09-11 12:38:41.013712: Current learning rate: 0.0003 
+2024-09-11 12:42:58.534469: train_loss -0.7779 
+2024-09-11 12:42:58.536972: val_loss -0.7875 
+2024-09-11 12:42:58.537909: Pseudo dice [0.9704, 0.9621, 0.9836, 0.9594, 0.8945, 0.9768, 0.9924] 
+2024-09-11 12:42:58.540106: Epoch time: 257.53 s 
+2024-09-11 12:43:02.142568:  
+2024-09-11 12:43:02.156768: Epoch 981 
+2024-09-11 12:43:02.157645: Current learning rate: 0.00028 
+2024-09-11 12:47:35.863388: train_loss -0.7696 
+2024-09-11 12:47:35.886685: val_loss -0.75 
+2024-09-11 12:47:35.888756: Pseudo dice [0.9696, 0.9609, 0.9835, 0.9568, 0.894, 0.9767, 0.9891] 
+2024-09-11 12:47:35.890779: Epoch time: 273.73 s 
+2024-09-11 12:47:38.951373:  
+2024-09-11 12:47:38.964771: Epoch 982 
+2024-09-11 12:47:38.965686: Current learning rate: 0.00027 
+2024-09-11 12:52:09.932828: train_loss -0.7728 
+2024-09-11 12:52:09.950235: val_loss -0.7546 
+2024-09-11 12:52:09.951257: Pseudo dice [0.968, 0.962, 0.9826, 0.9546, 0.8989, 0.9774, 0.9918] 
+2024-09-11 12:52:09.952234: Epoch time: 270.99 s 
+2024-09-11 12:52:12.691525:  
+2024-09-11 12:52:12.692919: Epoch 983 
+2024-09-11 12:52:12.693814: Current learning rate: 0.00026 
+2024-09-11 12:56:31.792660: train_loss -0.761 
+2024-09-11 12:56:31.812747: val_loss -0.7779 
+2024-09-11 12:56:31.814341: Pseudo dice [0.9699, 0.961, 0.9838, 0.9548, 0.8876, 0.9769, 0.9908] 
+2024-09-11 12:56:31.815807: Epoch time: 259.11 s 
+2024-09-11 12:56:34.990519:  
+2024-09-11 12:56:35.004860: Epoch 984 
+2024-09-11 12:56:35.005823: Current learning rate: 0.00024 
+2024-09-11 13:00:49.403516: train_loss -0.7587 
+2024-09-11 13:00:49.426301: val_loss -0.7601 
+2024-09-11 13:00:49.427152: Pseudo dice [0.9708, 0.9611, 0.9822, 0.955, 0.901, 0.9774, 0.9934] 
+2024-09-11 13:00:49.427973: Epoch time: 254.42 s 
+2024-09-11 13:00:51.882875:  
+2024-09-11 13:00:51.884182: Epoch 985 
+2024-09-11 13:00:51.885120: Current learning rate: 0.00023 
+2024-09-11 13:05:00.649125: train_loss -0.7706 
+2024-09-11 13:05:00.674364: val_loss -0.7683 
+2024-09-11 13:05:00.675431: Pseudo dice [0.9676, 0.9615, 0.9797, 0.9555, 0.8948, 0.9734, 0.992] 
+2024-09-11 13:05:00.676332: Epoch time: 248.77 s 
+2024-09-11 13:05:05.292860:  
+2024-09-11 13:05:05.294160: Epoch 986 
+2024-09-11 13:05:05.295311: Current learning rate: 0.00021 
+2024-09-11 13:09:35.803567: train_loss -0.7709 
+2024-09-11 13:09:35.811006: val_loss -0.7671 
+2024-09-11 13:09:35.811932: Pseudo dice [0.97, 0.9639, 0.9815, 0.9574, 0.8891, 0.9763, 0.9924] 
+2024-09-11 13:09:35.814192: Epoch time: 270.52 s 
+2024-09-11 13:09:37.981429:  
+2024-09-11 13:09:37.982648: Epoch 987 
+2024-09-11 13:09:37.983451: Current learning rate: 0.0002 
+2024-09-11 13:13:50.922006: train_loss -0.7707 
+2024-09-11 13:13:50.924616: val_loss -0.7724 
+2024-09-11 13:13:50.925687: Pseudo dice [0.9681, 0.9634, 0.9795, 0.9578, 0.8913, 0.9781, 0.9922] 
+2024-09-11 13:13:50.926837: Epoch time: 252.95 s 
+2024-09-11 13:13:56.565234:  
+2024-09-11 13:13:56.566260: Epoch 988 
+2024-09-11 13:13:56.567060: Current learning rate: 0.00019 
+2024-09-11 13:17:59.180007: train_loss -0.78 
+2024-09-11 13:17:59.194244: val_loss -0.7725 
+2024-09-11 13:17:59.195503: Pseudo dice [0.9687, 0.9621, 0.9806, 0.9589, 0.9049, 0.9783, 0.9932] 
+2024-09-11 13:17:59.196585: Epoch time: 242.62 s 
+2024-09-11 13:18:02.097016:  
+2024-09-11 13:18:02.108839: Epoch 989 
+2024-09-11 13:18:02.109847: Current learning rate: 0.00017 
+2024-09-11 13:22:25.657456: train_loss -0.7566 
+2024-09-11 13:22:25.673336: val_loss -0.7746 
+2024-09-11 13:22:25.674519: Pseudo dice [0.9679, 0.9606, 0.9803, 0.9556, 0.8973, 0.9742, 0.9904] 
+2024-09-11 13:22:25.675690: Epoch time: 263.56 s 
+2024-09-11 13:22:29.810372:  
+2024-09-11 13:22:29.811426: Epoch 990 
+2024-09-11 13:22:29.812312: Current learning rate: 0.00016 
+2024-09-11 13:26:49.777784: train_loss -0.7779 
+2024-09-11 13:26:49.780258: val_loss -0.7268 
+2024-09-11 13:26:49.781380: Pseudo dice [0.9713, 0.964, 0.9824, 0.9603, 0.8981, 0.9789, 0.9924] 
+2024-09-11 13:26:49.782323: Epoch time: 259.97 s 
+2024-09-11 13:26:52.965548:  
+2024-09-11 13:26:52.972050: Epoch 991 
+2024-09-11 13:26:52.973118: Current learning rate: 0.00014 
+2024-09-11 13:31:26.584204: train_loss -0.7749 
+2024-09-11 13:31:26.586752: val_loss -0.7458 
+2024-09-11 13:31:26.587821: Pseudo dice [0.9687, 0.9635, 0.9823, 0.9568, 0.8919, 0.9771, 0.992] 
+2024-09-11 13:31:26.588687: Epoch time: 273.62 s 
+2024-09-11 13:31:29.276920:  
+2024-09-11 13:31:29.278246: Epoch 992 
+2024-09-11 13:31:29.279256: Current learning rate: 0.00013 
+2024-09-11 13:36:11.480074: train_loss -0.7656 
+2024-09-11 13:36:11.502034: val_loss -0.7624 
+2024-09-11 13:36:11.503003: Pseudo dice [0.966, 0.9606, 0.9797, 0.9545, 0.8984, 0.9764, 0.9927] 
+2024-09-11 13:36:11.503813: Epoch time: 282.21 s 
+2024-09-11 13:36:15.128443:  
+2024-09-11 13:36:15.129680: Epoch 993 
+2024-09-11 13:36:15.130535: Current learning rate: 0.00011 
+2024-09-11 13:40:28.346234: train_loss -0.7641 
+2024-09-11 13:40:28.348825: val_loss -0.7733 
+2024-09-11 13:40:28.349958: Pseudo dice [0.9717, 0.9625, 0.9852, 0.963, 0.9027, 0.9807, 0.993] 
+2024-09-11 13:40:28.350958: Epoch time: 253.22 s 
+2024-09-11 13:40:31.697006:  
+2024-09-11 13:40:31.712900: Epoch 994 
+2024-09-11 13:40:31.714020: Current learning rate: 0.0001 
+2024-09-11 13:45:00.623549: train_loss -0.7641 
+2024-09-11 13:45:00.638420: val_loss -0.7924 
+2024-09-11 13:45:00.639381: Pseudo dice [0.9691, 0.965, 0.9832, 0.9594, 0.8998, 0.9779, 0.9914] 
+2024-09-11 13:45:00.640342: Epoch time: 268.93 s 
+2024-09-11 13:45:03.684332:  
+2024-09-11 13:45:03.685490: Epoch 995 
+2024-09-11 13:45:03.686527: Current learning rate: 8e-05 
+2024-09-11 13:49:33.707000: train_loss -0.7634 
+2024-09-11 13:49:33.709461: val_loss -0.7704 
+2024-09-11 13:49:33.710348: Pseudo dice [0.9696, 0.9627, 0.9828, 0.9535, 0.894, 0.9764, 0.9915] 
+2024-09-11 13:49:33.711191: Epoch time: 270.03 s 
+2024-09-11 13:49:36.740651:  
+2024-09-11 13:49:36.741695: Epoch 996 
+2024-09-11 13:49:36.742584: Current learning rate: 7e-05 
+2024-09-11 13:53:35.389385: train_loss -0.776 
+2024-09-11 13:53:35.408181: val_loss -0.7619 
+2024-09-11 13:53:35.409313: Pseudo dice [0.9703, 0.9635, 0.9824, 0.9599, 0.9043, 0.9798, 0.9936] 
+2024-09-11 13:53:35.410180: Epoch time: 238.65 s 
+2024-09-11 13:53:35.411024: Yayy! New best EMA pseudo Dice: 0.9627 
+2024-09-11 13:53:41.007111:  
+2024-09-11 13:53:41.008262: Epoch 997 
+2024-09-11 13:53:41.009265: Current learning rate: 5e-05 
+2024-09-11 13:57:55.062960: train_loss -0.762 
+2024-09-11 13:57:55.065319: val_loss -0.769 
+2024-09-11 13:57:55.066213: Pseudo dice [0.9666, 0.9613, 0.9799, 0.9593, 0.8982, 0.9764, 0.992] 
+2024-09-11 13:57:55.068638: Epoch time: 254.06 s 
+2024-09-11 13:57:58.181869:  
+2024-09-11 13:57:58.183187: Epoch 998 
+2024-09-11 13:57:58.184292: Current learning rate: 4e-05 
+2024-09-11 14:02:22.507598: train_loss -0.7548 
+2024-09-11 14:02:22.510154: val_loss -0.7943 
+2024-09-11 14:02:22.511076: Pseudo dice [0.9689, 0.9635, 0.9822, 0.9576, 0.8938, 0.9766, 0.9905] 
+2024-09-11 14:02:22.511816: Epoch time: 264.33 s 
+2024-09-11 14:02:25.065914:  
+2024-09-11 14:02:25.067001: Epoch 999 
+2024-09-11 14:02:25.067907: Current learning rate: 2e-05 
+2024-09-11 14:06:40.736854: train_loss -0.7754 
+2024-09-11 14:06:40.765770: val_loss -0.7543 
+2024-09-11 14:06:40.766656: Pseudo dice [0.9669, 0.9605, 0.98, 0.9568, 0.9003, 0.9761, 0.993] 
+2024-09-11 14:06:40.767414: Epoch time: 255.68 s 
+2024-09-11 14:06:47.314023: Training done. 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_11_14_42_12.txt

@@ -0,0 +1,692 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-11 14:42:20.747971: Using torch.compile... 
+2024-09-11 14:42:33.838852: do_dummy_2d_data_aug: False 
+2024-09-11 14:42:33.844605: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-11 14:42:33.846171: The split file contains 5 splits. 
+2024-09-11 14:42:33.846608: Desired fold for training: 0 
+2024-09-11 14:42:33.846963: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-11 14:42:47.569272: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-11 14:42:58.177481: Training done. 
+2024-09-11 14:42:58.400854: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-11 14:42:58.403152: The split file contains 5 splits. 
+2024-09-11 14:42:58.403682: Desired fold for training: 0 
+2024-09-11 14:42:58.404181: This split has 1324 training and 332 validation cases. 
+2024-09-11 14:42:58.411775: predicting saros-023 
+2024-09-11 14:42:58.711264: saros-023, shape torch.Size([1, 263, 234, 234]), rank 0 
+2024-09-11 14:46:07.585252: predicting saros-030 
+2024-09-11 14:46:08.107234: saros-030, shape torch.Size([1, 323, 260, 260]), rank 0 
+2024-09-11 14:46:32.492860: predicting saros-031 
+2024-09-11 14:46:32.656633: saros-031, shape torch.Size([1, 123, 253, 253]), rank 0 
+2024-09-11 14:46:51.526160: predicting saros-033 
+2024-09-11 14:46:51.962795: saros-033, shape torch.Size([1, 290, 253, 253]), rank 0 
+2024-09-11 14:47:11.013598: predicting saros-039 
+2024-09-11 14:47:11.522647: saros-039, shape torch.Size([1, 327, 240, 240]), rank 0 
+2024-09-11 14:47:44.277232: predicting saros-044 
+2024-09-11 14:47:44.611222: saros-044, shape torch.Size([1, 193, 251, 251]), rank 0 
+2024-09-11 14:48:10.728248: predicting saros-049 
+2024-09-11 14:48:11.052203: saros-049, shape torch.Size([1, 227, 252, 252]), rank 0 
+2024-09-11 14:48:38.607302: predicting saros-060 
+2024-09-11 14:48:38.908901: saros-060, shape torch.Size([1, 170, 253, 253]), rank 0 
+2024-09-11 14:48:42.038875: predicting saros-063 
+2024-09-11 14:48:42.305028: saros-063, shape torch.Size([1, 207, 240, 240]), rank 0 
+2024-09-11 14:49:13.900374: predicting saros-065 
+2024-09-11 14:49:14.241746: saros-065, shape torch.Size([1, 213, 269, 269]), rank 0 
+2024-09-11 14:49:46.228956: predicting saros-066 
+2024-09-11 14:49:46.401727: saros-066, shape torch.Size([1, 100, 259, 259]), rank 0 
+2024-09-11 14:50:16.250091: predicting saros-067 
+2024-09-11 14:50:16.505183: saros-067, shape torch.Size([1, 203, 240, 240]), rank 0 
+2024-09-11 14:50:32.179102: predicting saros-070 
+2024-09-11 14:50:32.577975: saros-070, shape torch.Size([1, 207, 293, 293]), rank 0 
+2024-09-11 14:51:08.727600: predicting saros-072 
+2024-09-11 14:51:08.992149: saros-072, shape torch.Size([1, 137, 307, 307]), rank 0 
+2024-09-11 14:51:35.876575: predicting saros-076 
+2024-09-11 14:51:36.155357: saros-076, shape torch.Size([1, 140, 307, 307]), rank 0 
+2024-09-11 14:51:41.046729: predicting saros-078 
+2024-09-11 14:51:41.484664: saros-078, shape torch.Size([1, 310, 253, 253]), rank 0 
+2024-09-11 14:52:00.709760: predicting saros-086 
+2024-09-11 14:52:01.185341: saros-086, shape torch.Size([1, 313, 266, 266]), rank 0 
+2024-09-11 14:54:20.988514: predicting saros-096 
+2024-09-11 14:54:21.255213: saros-096, shape torch.Size([1, 207, 253, 253]), rank 0 
+2024-09-11 14:54:26.290879: predicting saros-098 
+2024-09-11 14:54:26.545714: saros-098, shape torch.Size([1, 207, 240, 240]), rank 0 
+2024-09-11 14:54:31.770304: predicting saros-110 
+2024-09-11 14:54:32.415299: saros-110, shape torch.Size([1, 320, 307, 307]), rank 0 
+2024-09-11 14:54:46.848687: predicting saros-120 
+2024-09-11 14:54:47.471577: saros-120, shape torch.Size([1, 290, 289, 289]), rank 0 
+2024-09-11 14:55:15.518659: predicting saros-136 
+2024-09-11 14:55:15.749262: saros-136, shape torch.Size([1, 137, 267, 267]), rank 0 
+2024-09-11 14:55:34.816509: predicting saros-137 
+2024-09-11 14:55:35.173473: saros-137, shape torch.Size([1, 287, 240, 240]), rank 0 
+2024-09-11 14:55:39.918804: predicting saros-139 
+2024-09-11 14:55:40.038218: saros-139, shape torch.Size([1, 127, 200, 200]), rank 0 
+2024-09-11 14:55:59.061361: predicting saros-141 
+2024-09-11 14:55:59.494850: saros-141, shape torch.Size([1, 183, 333, 333]), rank 0 
+2024-09-11 14:56:23.314246: predicting saros-168 
+2024-09-11 14:56:23.751375: saros-168, shape torch.Size([1, 233, 283, 283]), rank 0 
+2024-09-11 14:56:47.192683: predicting saros-174 
+2024-09-11 14:56:47.532131: saros-174, shape torch.Size([1, 153, 333, 333]), rank 0 
+2024-09-11 14:56:52.469497: predicting saros-192 
+2024-09-11 14:56:53.140015: saros-192, shape torch.Size([1, 303, 320, 320]), rank 0 
+2024-09-11 14:57:24.058341: predicting saros-196 
+2024-09-11 14:57:24.398132: saros-196, shape torch.Size([1, 137, 333, 333]), rank 0 
+2024-09-11 14:57:30.045830: predicting saros-198 
+2024-09-11 14:57:30.899012: saros-198, shape torch.Size([1, 367, 333, 333]), rank 0 
+2024-09-11 14:57:45.093846: predicting saros-204 
+2024-09-11 14:57:45.690043: saros-204, shape torch.Size([1, 343, 273, 273]), rank 0 
+2024-09-11 14:58:01.461616: predicting saros-208 
+2024-09-11 14:58:01.796349: saros-208, shape torch.Size([1, 143, 320, 320]), rank 0 
+2024-09-11 14:58:06.388645: predicting saros-209 
+2024-09-11 14:58:06.727299: saros-209, shape torch.Size([1, 153, 320, 320]), rank 0 
+2024-09-11 14:58:11.707617: predicting saros-210 
+2024-09-11 14:58:12.348337: saros-210, shape torch.Size([1, 337, 287, 287]), rank 0 
+2024-09-11 14:58:19.737812: predicting saros-211 
+2024-09-11 14:58:19.927122: saros-211, shape torch.Size([1, 143, 239, 239]), rank 0 
+2024-09-11 14:58:23.014731: predicting saros-213 
+2024-09-11 14:58:23.662251: saros-213, shape torch.Size([1, 340, 275, 275]), rank 0 
+2024-09-11 14:58:30.979269: predicting saros-215 
+2024-09-11 14:58:31.474462: saros-215, shape torch.Size([1, 290, 277, 277]), rank 0 
+2024-09-11 14:58:38.669736: predicting saros-218 
+2024-09-11 14:58:39.353624: saros-218, shape torch.Size([1, 303, 260, 260]), rank 0 
+2024-09-11 14:58:46.127944: predicting saros-227 
+2024-09-11 14:58:46.384503: saros-227, shape torch.Size([1, 160, 251, 251]), rank 0 
+2024-09-11 14:58:49.169574: predicting saros-231 
+2024-09-11 14:58:49.992952: saros-231, shape torch.Size([1, 343, 333, 333]), rank 0 
+2024-09-11 14:59:05.375624: predicting saros-235 
+2024-09-11 14:59:06.046223: saros-235, shape torch.Size([1, 323, 298, 298]), rank 0 
+2024-09-11 14:59:21.203490: predicting saros-239 
+2024-09-11 14:59:21.371104: saros-239, shape torch.Size([1, 110, 253, 253]), rank 0 
+2024-09-11 14:59:24.129834: predicting saros-247 
+2024-09-11 14:59:24.328514: saros-247, shape torch.Size([1, 137, 267, 267]), rank 0 
+2024-09-11 14:59:27.839057: predicting saros-260 
+2024-09-11 14:59:28.460262: saros-260, shape torch.Size([1, 337, 300, 300]), rank 0 
+2024-09-11 14:59:44.042921: predicting saros-265 
+2024-09-11 14:59:44.635044: saros-265, shape torch.Size([1, 297, 321, 321]), rank 0 
+2024-09-11 14:59:59.500978: predicting saros-280 
+2024-09-11 14:59:59.985846: saros-280, shape torch.Size([1, 300, 264, 264]), rank 0 
+2024-09-11 15:00:08.531799: predicting saros-291 
+2024-09-11 15:00:08.920744: saros-291, shape torch.Size([1, 300, 233, 233]), rank 0 
+2024-09-11 15:00:16.062977: predicting saros-294 
+2024-09-11 15:00:16.428468: saros-294, shape torch.Size([1, 147, 315, 315]), rank 0 
+2024-09-11 15:00:21.959772: predicting saros-298 
+2024-09-11 15:00:22.613900: saros-298, shape torch.Size([1, 307, 313, 313]), rank 0 
+2024-09-11 15:00:38.443233: predicting saros-299 
+2024-09-11 15:00:39.037169: saros-299, shape torch.Size([1, 300, 296, 296]), rank 0 
+2024-09-11 15:00:54.487052: predicting saros-302 
+2024-09-11 15:00:54.673365: saros-302, shape torch.Size([1, 160, 240, 240]), rank 0 
+2024-09-11 15:00:58.248307: predicting saros-314 
+2024-09-11 15:00:58.492154: saros-314, shape torch.Size([1, 220, 223, 223]), rank 0 
+2024-09-11 15:01:03.635339: predicting saros-326 
+2024-09-11 15:01:04.141603: saros-326, shape torch.Size([1, 223, 333, 333]), rank 0 
+2024-09-11 15:01:15.182584: predicting saros-328 
+2024-09-11 15:01:15.504896: saros-328, shape torch.Size([1, 220, 240, 240]), rank 0 
+2024-09-11 15:01:20.379667: predicting saros-333 
+2024-09-11 15:01:20.807786: saros-333, shape torch.Size([1, 213, 333, 333]), rank 0 
+2024-09-11 15:01:31.865210: predicting saros-338 
+2024-09-11 15:01:32.147314: saros-338, shape torch.Size([1, 220, 227, 227]), rank 0 
+2024-09-11 15:01:37.638852: predicting saros-344 
+2024-09-11 15:01:37.910794: saros-344, shape torch.Size([1, 213, 226, 226]), rank 0 
+2024-09-11 15:01:43.978094: predicting saros-346 
+2024-09-11 15:01:44.230205: saros-346, shape torch.Size([1, 180, 222, 222]), rank 0 
+2024-09-11 15:01:47.762891: predicting saros-352 
+2024-09-11 15:01:48.078210: saros-352, shape torch.Size([1, 200, 254, 254]), rank 0 
+2024-09-11 15:01:53.666324: predicting saros-361 
+2024-09-11 15:01:54.325386: saros-361, shape torch.Size([1, 273, 333, 333]), rank 0 
+2024-09-11 15:02:05.712353: predicting saros-363 
+2024-09-11 15:02:05.918192: saros-363, shape torch.Size([1, 193, 227, 227]), rank 0 
+2024-09-11 15:02:10.725890: predicting saros-376 
+2024-09-11 15:02:11.212141: saros-376, shape torch.Size([1, 257, 313, 313]), rank 0 
+2024-09-11 15:02:22.002859: predicting saros-381 
+2024-09-11 15:02:22.187593: saros-381, shape torch.Size([1, 150, 240, 240]), rank 0 
+2024-09-11 15:02:25.770661: predicting saros-390 
+2024-09-11 15:02:26.162654: saros-390, shape torch.Size([1, 187, 333, 333]), rank 0 
+2024-09-11 15:02:31.920219: predicting saros-396 
+2024-09-11 15:02:32.132850: saros-396, shape torch.Size([1, 187, 195, 195]), rank 0 
+2024-09-11 15:02:35.278546: predicting saros-398 
+2024-09-11 15:02:35.718767: saros-398, shape torch.Size([1, 183, 333, 333]), rank 0 
+2024-09-11 15:02:42.637803: predicting saros-405 
+2024-09-11 15:02:42.916741: saros-405, shape torch.Size([1, 200, 240, 240]), rank 0 
+2024-09-11 15:02:48.115961: predicting saros-412 
+2024-09-11 15:02:48.504338: saros-412, shape torch.Size([1, 203, 281, 281]), rank 0 
+2024-09-11 15:02:54.272602: predicting saros-422 
+2024-09-11 15:02:54.569873: saros-422, shape torch.Size([1, 200, 248, 248]), rank 0 
+2024-09-11 15:02:59.356724: predicting saros-428 
+2024-09-11 15:02:59.704739: saros-428, shape torch.Size([1, 220, 267, 267]), rank 0 
+2024-09-11 15:03:04.310900: predicting saros-431 
+2024-09-11 15:03:04.488351: saros-431, shape torch.Size([1, 207, 181, 181]), rank 0 
+2024-09-11 15:03:05.973466: predicting saros-435 
+2024-09-11 15:03:06.275788: saros-435, shape torch.Size([1, 210, 266, 266]), rank 0 
+2024-09-11 15:03:11.526795: predicting saros-447 
+2024-09-11 15:03:11.772806: saros-447, shape torch.Size([1, 207, 210, 210]), rank 0 
+2024-09-11 15:03:16.665877: predicting saros-448 
+2024-09-11 15:03:17.153212: saros-448, shape torch.Size([1, 227, 316, 316]), rank 0 
+2024-09-11 15:03:27.096136: predicting saros-449 
+2024-09-11 15:03:27.452766: saros-449, shape torch.Size([1, 220, 227, 227]), rank 0 
+2024-09-11 15:03:32.883471: predicting saros-453 
+2024-09-11 15:03:33.111802: saros-453, shape torch.Size([1, 210, 204, 204]), rank 0 
+2024-09-11 15:03:38.511441: predicting saros-465 
+2024-09-11 15:03:39.000205: saros-465, shape torch.Size([1, 253, 287, 287]), rank 0 
+2024-09-11 15:03:43.852454: predicting saros-481 
+2024-09-11 15:03:44.351125: saros-481, shape torch.Size([1, 253, 287, 287]), rank 0 
+2024-09-11 15:03:49.380824: predicting saros-485 
+2024-09-11 15:03:49.857776: saros-485, shape torch.Size([1, 260, 269, 269]), rank 0 
+2024-09-11 15:03:56.261278: predicting saros-493 
+2024-09-11 15:03:56.909898: saros-493, shape torch.Size([1, 263, 333, 333]), rank 0 
+2024-09-11 15:04:07.173815: predicting saros-494 
+2024-09-11 15:04:07.374961: saros-494, shape torch.Size([1, 253, 183, 183]), rank 0 
+2024-09-11 15:04:08.827655: predicting saros-512 
+2024-09-11 15:04:09.225861: saros-512, shape torch.Size([1, 220, 257, 257]), rank 0 
+2024-09-11 15:04:13.765269: predicting saros-516 
+2024-09-11 15:04:14.030460: saros-516, shape torch.Size([1, 187, 240, 240]), rank 0 
+2024-09-11 15:04:16.630085: predicting saros-519 
+2024-09-11 15:04:17.313325: saros-519, shape torch.Size([1, 283, 333, 333]), rank 0 
+2024-09-11 15:04:27.947069: predicting saros-522 
+2024-09-11 15:04:28.277662: saros-522, shape torch.Size([1, 233, 237, 237]), rank 0 
+2024-09-11 15:04:33.416668: predicting saros-525 
+2024-09-11 15:04:33.825752: saros-525, shape torch.Size([1, 240, 267, 267]), rank 0 
+2024-09-11 15:04:38.792591: predicting saros-533 
+2024-09-11 15:04:39.120663: saros-533, shape torch.Size([1, 233, 244, 244]), rank 0 
+2024-09-11 15:04:44.244573: predicting saros-536 
+2024-09-11 15:04:44.638949: saros-536, shape torch.Size([1, 260, 277, 277]), rank 0 
+2024-09-11 15:04:50.390510: predicting saros-542 
+2024-09-11 15:04:50.764847: saros-542, shape torch.Size([1, 220, 267, 267]), rank 0 
+2024-09-11 15:04:56.592160: predicting saros-543 
+2024-09-11 15:04:56.822675: saros-543, shape torch.Size([1, 147, 253, 253]), rank 0 
+2024-09-11 15:05:00.157324: predicting saros-545 
+2024-09-11 15:05:00.458949: saros-545, shape torch.Size([1, 233, 232, 232]), rank 0 
+2024-09-11 15:05:06.334726: predicting saros-549 
+2024-09-11 15:05:06.616570: saros-549, shape torch.Size([1, 243, 222, 222]), rank 0 
+2024-09-11 15:05:11.859503: predicting saros-568 
+2024-09-11 15:05:12.251145: saros-568, shape torch.Size([1, 260, 275, 275]), rank 0 
+2024-09-11 15:05:17.894793: predicting saros-570 
+2024-09-11 15:05:18.687936: saros-570, shape torch.Size([1, 353, 333, 333]), rank 0 
+2024-09-11 15:05:34.583586: predicting saros-571 
+2024-09-11 15:05:35.094753: saros-571, shape torch.Size([1, 240, 307, 307]), rank 0 
+2024-09-11 15:05:46.181615: predicting saros-575 
+2024-09-11 15:05:46.569641: saros-575, shape torch.Size([1, 253, 250, 250]), rank 0 
+2024-09-11 15:05:52.140265: predicting saros-582 
+2024-09-11 15:05:52.393627: saros-582, shape torch.Size([1, 160, 260, 260]), rank 0 
+2024-09-11 15:05:55.865710: predicting saros-583 
+2024-09-11 15:05:56.413641: saros-583, shape torch.Size([1, 273, 333, 333]), rank 0 
+2024-09-11 15:06:07.978706: predicting saros-588 
+2024-09-11 15:06:08.493527: saros-588, shape torch.Size([1, 240, 307, 307]), rank 0 
+2024-09-11 15:06:19.387559: predicting saros-589 
+2024-09-11 15:06:20.103736: saros-589, shape torch.Size([1, 353, 333, 333]), rank 0 
+2024-09-11 15:06:35.872642: predicting saros-603 
+2024-09-11 15:06:38.222896: saros-603, shape torch.Size([1, 737, 400, 400]), rank 0 
+2024-09-11 15:07:39.184692: predicting saros-605 
+2024-09-11 15:07:40.433363: saros-605, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:08:06.194476: predicting saros-615 
+2024-09-11 15:08:07.259978: saros-615, shape torch.Size([1, 497, 333, 333]), rank 0 
+2024-09-11 15:08:32.826353: predicting saros-625 
+2024-09-11 15:08:34.082288: saros-625, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:08:59.261874: predicting saros-631 
+2024-09-11 15:09:00.923403: saros-631, shape torch.Size([1, 660, 333, 333]), rank 0 
+2024-09-11 15:09:30.886213: predicting saros-649 
+2024-09-11 15:09:32.110008: saros-649, shape torch.Size([1, 547, 333, 333]), rank 0 
+2024-09-11 15:09:56.956487: predicting saros-652 
+2024-09-11 15:09:58.246038: saros-652, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:10:23.248206: predicting saros-657 
+2024-09-11 15:10:24.282265: saros-657, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-11 15:10:45.049803: predicting saros-658 
+2024-09-11 15:10:46.135105: saros-658, shape torch.Size([1, 497, 333, 333]), rank 0 
+2024-09-11 15:11:11.313942: predicting saros-664 
+2024-09-11 15:11:13.063085: saros-664, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-11 15:11:47.779721: predicting saros-668 
+2024-09-11 15:11:48.994719: saros-668, shape torch.Size([1, 483, 333, 333]), rank 0 
+2024-09-11 15:12:14.558729: predicting saros-669 
+2024-09-11 15:12:15.789567: saros-669, shape torch.Size([1, 567, 333, 333]), rank 0 
+2024-09-11 15:12:40.766508: predicting saros-673 
+2024-09-11 15:12:42.240496: saros-673, shape torch.Size([1, 593, 333, 333]), rank 0 
+2024-09-11 15:13:12.722158: predicting saros-675 
+2024-09-11 15:13:14.029688: saros-675, shape torch.Size([1, 547, 333, 333]), rank 0 
+2024-09-11 15:13:39.377151: predicting saros-688 
+2024-09-11 15:13:40.641755: saros-688, shape torch.Size([1, 487, 333, 333]), rank 0 
+2024-09-11 15:14:06.101424: predicting saros-693 
+2024-09-11 15:14:07.382483: saros-693, shape torch.Size([1, 567, 333, 333]), rank 0 
+2024-09-11 15:14:32.650226: predicting saros-709 
+2024-09-11 15:14:34.356812: saros-709, shape torch.Size([1, 653, 333, 333]), rank 0 
+2024-09-11 15:15:04.165902: predicting saros-711 
+2024-09-11 15:15:05.553644: saros-711, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:15:30.386236: predicting saros-731 
+2024-09-11 15:15:31.995224: saros-731, shape torch.Size([1, 660, 333, 333]), rank 0 
+2024-09-11 15:16:01.599917: predicting saros-734 
+2024-09-11 15:16:02.951835: saros-734, shape torch.Size([1, 593, 333, 333]), rank 0 
+2024-09-11 15:16:33.233624: predicting saros-739 
+2024-09-11 15:16:34.202432: saros-739, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-11 15:16:54.770679: predicting saros-741 
+2024-09-11 15:16:56.220111: saros-741, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:17:22.227511: predicting saros-743 
+2024-09-11 15:17:23.684939: saros-743, shape torch.Size([1, 593, 333, 333]), rank 0 
+2024-09-11 15:17:53.209739: predicting saros-744 
+2024-09-11 15:17:54.373634: saros-744, shape torch.Size([1, 493, 333, 333]), rank 0 
+2024-09-11 15:18:19.723549: predicting saros-745 
+2024-09-11 15:18:21.149374: saros-745, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:18:46.417352: predicting saros-751 
+2024-09-11 15:18:47.903865: saros-751, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-11 15:19:22.707343: predicting saros-756 
+2024-09-11 15:19:24.276844: saros-756, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-11 15:19:59.658061: predicting saros-774 
+2024-09-11 15:20:00.908715: saros-774, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:20:26.821769: predicting saros-777 
+2024-09-11 15:20:27.804826: saros-777, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-11 15:20:48.010694: predicting saros-778 
+2024-09-11 15:20:49.561494: saros-778, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-11 15:21:23.672243: predicting saros-783 
+2024-09-11 15:21:26.080562: saros-783, shape torch.Size([1, 1267, 333, 333]), rank 0 
+2024-09-11 15:22:29.986797: predicting saros-789 
+2024-09-11 15:22:31.631210: saros-789, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-11 15:23:06.613932: predicting saros-794 
+2024-09-11 15:23:07.916623: saros-794, shape torch.Size([1, 593, 333, 333]), rank 0 
+2024-09-11 15:23:37.703503: predicting saros-800 
+2024-09-11 15:23:38.703450: saros-800, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-11 15:23:59.160469: predicting saros-801 
+2024-09-11 15:24:00.503760: saros-801, shape torch.Size([1, 660, 333, 333]), rank 0 
+2024-09-11 15:24:30.194169: predicting saros-807 
+2024-09-11 15:24:31.796144: saros-807, shape torch.Size([1, 677, 333, 333]), rank 0 
+2024-09-11 15:25:06.048196: predicting saros-809 
+2024-09-11 15:25:07.444818: saros-809, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-11 15:25:42.114877: predicting saros-820 
+2024-09-11 15:25:44.836538: saros-820, shape torch.Size([1, 730, 467, 467]), rank 0 
+2024-09-11 15:26:46.532171: predicting saros-826 
+2024-09-11 15:26:48.053841: saros-826, shape torch.Size([1, 660, 333, 333]), rank 0 
+2024-09-11 15:27:17.560173: predicting saros-827 
+2024-09-11 15:27:18.873240: saros-827, shape torch.Size([1, 647, 333, 333]), rank 0 
+2024-09-11 15:27:49.600133: predicting saros-829 
+2024-09-11 15:27:50.600619: saros-829, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-11 15:28:11.825669: predicting saros-832 
+2024-09-11 15:28:14.032345: saros-832, shape torch.Size([1, 647, 467, 467]), rank 0 
+2024-09-11 15:29:06.420720: predicting saros-834 
+2024-09-11 15:29:07.953499: saros-834, shape torch.Size([1, 660, 333, 333]), rank 0 
+2024-09-11 15:29:38.306506: predicting saros-837 
+2024-09-11 15:29:39.373389: saros-837, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-11 15:29:59.440566: predicting saros-843 
+2024-09-11 15:30:00.869207: saros-843, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-11 15:30:34.977491: predicting saros-857 
+2024-09-11 15:30:36.437589: saros-857, shape torch.Size([1, 653, 333, 333]), rank 0 
+2024-09-11 15:31:06.544766: predicting saros-874 
+2024-09-11 15:31:07.803932: saros-874, shape torch.Size([1, 603, 333, 333]), rank 0 
+2024-09-11 15:31:38.290126: predicting saros-885 
+2024-09-11 15:31:39.678406: saros-885, shape torch.Size([1, 650, 333, 333]), rank 0 
+2024-09-11 15:32:09.774080: predicting saros-891 
+2024-09-11 15:32:11.095427: saros-891, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-11 15:32:36.545115: predicting saros-895 
+2024-09-11 15:32:37.866714: saros-895, shape torch.Size([1, 647, 333, 333]), rank 0 
+2024-09-11 15:33:08.869126: predicting tseg-s0009 
+2024-09-11 15:33:08.932684: tseg-s0009, shape torch.Size([1, 185, 119, 119]), rank 0 
+2024-09-11 15:33:10.911857: predicting tseg-s0010 
+2024-09-11 15:33:11.232141: tseg-s0010, shape torch.Size([1, 283, 259, 259]), rank 0 
+2024-09-11 15:33:17.200051: predicting tseg-s0022 
+2024-09-11 15:33:17.632990: tseg-s0022, shape torch.Size([1, 250, 295, 295]), rank 0 
+2024-09-11 15:33:28.188624: predicting tseg-s0024 
+2024-09-11 15:33:28.933658: tseg-s0024, shape torch.Size([1, 486, 287, 287]), rank 0 
+2024-09-11 15:33:41.219859: predicting tseg-s0030 
+2024-09-11 15:33:41.740979: tseg-s0030, shape torch.Size([1, 317, 309, 309]), rank 0 
+2024-09-11 15:33:57.521790: predicting tseg-s0034 
+2024-09-11 15:33:57.974566: tseg-s0034, shape torch.Size([1, 283, 308, 308]), rank 0 
+2024-09-11 15:34:08.736745: predicting tseg-s0035 
+2024-09-11 15:34:08.980199: tseg-s0035, shape torch.Size([1, 145, 320, 320]), rank 0 
+2024-09-11 15:34:14.839770: predicting tseg-s0036 
+2024-09-11 15:34:14.887512: tseg-s0036, shape torch.Size([1, 137, 120, 120]), rank 0 
+2024-09-11 15:34:15.916331: predicting tseg-s0049 
+2024-09-11 15:34:16.576978: tseg-s0049, shape torch.Size([1, 455, 221, 221]), rank 0 
+2024-09-11 15:34:25.696866: predicting tseg-s0050 
+2024-09-11 15:34:26.104031: tseg-s0050, shape torch.Size([1, 399, 182, 261]), rank 0 
+2024-09-11 15:34:31.666040: predicting tseg-s0052 
+2024-09-11 15:34:32.021051: tseg-s0052, shape torch.Size([1, 294, 249, 249]), rank 0 
+2024-09-11 15:34:39.045621: predicting tseg-s0059 
+2024-09-11 15:34:39.478890: tseg-s0059, shape torch.Size([1, 449, 233, 233]), rank 0 
+2024-09-11 15:34:48.498040: predicting tseg-s0072 
+2024-09-11 15:34:48.869492: tseg-s0072, shape torch.Size([1, 269, 286, 286]), rank 0 
+2024-09-11 15:34:54.404231: predicting tseg-s0073 
+2024-09-11 15:34:54.715248: tseg-s0073, shape torch.Size([1, 265, 244, 247]), rank 0 
+2024-09-11 15:34:59.783901: predicting tseg-s0076 
+2024-09-11 15:35:00.556227: tseg-s0076, shape torch.Size([1, 441, 319, 319]), rank 0 
+2024-09-11 15:35:21.171546: predicting tseg-s0088 
+2024-09-11 15:35:21.316465: tseg-s0088, shape torch.Size([1, 90, 311, 311]), rank 0 
+2024-09-11 15:35:27.636009: predicting tseg-s0089 
+2024-09-11 15:35:27.879429: tseg-s0089, shape torch.Size([1, 263, 221, 221]), rank 0 
+2024-09-11 15:35:33.770001: predicting tseg-s0100 
+2024-09-11 15:35:33.799414: tseg-s0100, shape torch.Size([1, 121, 91, 91]), rank 0 
+2024-09-11 15:35:34.952139: predicting tseg-s0101 
+2024-09-11 15:35:34.993347: tseg-s0101, shape torch.Size([1, 74, 119, 119]), rank 0 
+2024-09-11 15:35:36.863073: predicting tseg-s0129 
+2024-09-11 15:35:36.883281: tseg-s0129, shape torch.Size([1, 148, 111, 50]), rank 0 
+2024-09-11 15:35:38.670236: predicting tseg-s0136 
+2024-09-11 15:35:38.712862: tseg-s0136, shape torch.Size([1, 148, 111, 81]), rank 0 
+2024-09-11 15:35:40.221082: predicting tseg-s0151 
+2024-09-11 15:35:40.485752: tseg-s0151, shape torch.Size([1, 292, 184, 292]), rank 0 
+2024-09-11 15:35:46.419679: predicting tseg-s0152 
+2024-09-11 15:35:46.621781: tseg-s0152, shape torch.Size([1, 224, 233, 233]), rank 0 
+2024-09-11 15:35:51.888277: predicting tseg-s0157 
+2024-09-11 15:35:52.397153: tseg-s0157, shape torch.Size([1, 261, 314, 314]), rank 0 
+2024-09-11 15:36:03.640614: predicting tseg-s0165 
+2024-09-11 15:36:03.794891: tseg-s0165, shape torch.Size([1, 91, 304, 304]), rank 0 
+2024-09-11 15:36:09.224602: predicting tseg-s0181 
+2024-09-11 15:36:09.287484: tseg-s0181, shape torch.Size([1, 52, 244, 244]), rank 0 
+2024-09-11 15:36:12.758029: predicting tseg-s0187 
+2024-09-11 15:36:12.827737: tseg-s0187, shape torch.Size([1, 125, 180, 180]), rank 0 
+2024-09-11 15:36:14.324458: predicting tseg-s0191 
+2024-09-11 15:36:14.400964: tseg-s0191, shape torch.Size([1, 338, 111, 98]), rank 0 
+2024-09-11 15:36:17.678600: predicting tseg-s0192 
+2024-09-11 15:36:17.886015: tseg-s0192, shape torch.Size([1, 296, 136, 239]), rank 0 
+2024-09-11 15:36:23.194876: predicting tseg-s0194 
+2024-09-11 15:36:23.445512: tseg-s0194, shape torch.Size([1, 279, 227, 227]), rank 0 
+2024-09-11 15:36:29.424992: predicting tseg-s0210 
+2024-09-11 15:36:30.011349: tseg-s0210, shape torch.Size([1, 317, 333, 333]), rank 0 
+2024-09-11 15:36:44.877178: predicting tseg-s0213 
+2024-09-11 15:36:45.046093: tseg-s0213, shape torch.Size([1, 267, 166, 267]), rank 0 
+2024-09-11 15:36:47.860050: predicting tseg-s0223 
+2024-09-11 15:36:48.070052: tseg-s0223, shape torch.Size([1, 191, 251, 251]), rank 0 
+2024-09-11 15:36:50.988376: predicting tseg-s0240 
+2024-09-11 15:36:51.344191: tseg-s0240, shape torch.Size([1, 204, 309, 309]), rank 0 
+2024-09-11 15:37:01.203655: predicting tseg-s0245 
+2024-09-11 15:37:01.342366: tseg-s0245, shape torch.Size([1, 95, 277, 277]), rank 0 
+2024-09-11 15:37:05.082696: predicting tseg-s0246 
+2024-09-11 15:37:05.163658: tseg-s0246, shape torch.Size([1, 77, 258, 258]), rank 0 
+2024-09-11 15:37:07.731127: predicting tseg-s0255 
+2024-09-11 15:37:08.096921: tseg-s0255, shape torch.Size([1, 306, 261, 261]), rank 0 
+2024-09-11 15:37:15.055928: predicting tseg-s0258 
+2024-09-11 15:37:15.182320: tseg-s0258, shape torch.Size([1, 127, 242, 242]), rank 0 
+2024-09-11 15:37:18.067078: predicting tseg-s0261 
+2024-09-11 15:37:18.215603: tseg-s0261, shape torch.Size([1, 191, 223, 223]), rank 0 
+2024-09-11 15:37:21.788229: predicting tseg-s0265 
+2024-09-11 15:37:21.847379: tseg-s0265, shape torch.Size([1, 44, 261, 261]), rank 0 
+2024-09-11 15:37:24.376139: predicting tseg-s0270 
+2024-09-11 15:37:24.529775: tseg-s0270, shape torch.Size([1, 91, 333, 333]), rank 0 
+2024-09-11 15:37:29.785848: predicting tseg-s0292 
+2024-09-11 15:37:29.870551: tseg-s0292, shape torch.Size([1, 233, 148, 148]), rank 0 
+2024-09-11 15:37:31.831834: predicting tseg-s0322 
+2024-09-11 15:37:32.187626: tseg-s0322, shape torch.Size([1, 209, 299, 299]), rank 0 
+2024-09-11 15:37:42.524143: predicting tseg-s0339 
+2024-09-11 15:37:42.830003: tseg-s0339, shape torch.Size([1, 281, 240, 240]), rank 0 
+2024-09-11 15:37:47.905285: predicting tseg-s0345 
+2024-09-11 15:37:48.573336: tseg-s0345, shape torch.Size([1, 476, 261, 238]), rank 0 
+2024-09-11 15:37:58.249831: predicting tseg-s0346 
+2024-09-11 15:37:58.440039: tseg-s0346, shape torch.Size([1, 241, 215, 215]), rank 0 
+2024-09-11 15:38:03.765691: predicting tseg-s0358 
+2024-09-11 15:38:04.328240: tseg-s0358, shape torch.Size([1, 523, 255, 255]), rank 0 
+2024-09-11 15:38:16.353117: predicting tseg-s0362 
+2024-09-11 15:38:17.020844: tseg-s0362, shape torch.Size([1, 417, 285, 285]), rank 0 
+2024-09-11 15:38:27.001556: predicting tseg-s0365 
+2024-09-11 15:38:27.278321: tseg-s0365, shape torch.Size([1, 222, 257, 257]), rank 0 
+2024-09-11 15:38:32.930881: predicting tseg-s0367 
+2024-09-11 15:38:33.140398: tseg-s0367, shape torch.Size([1, 205, 212, 212]), rank 0 
+2024-09-11 15:38:38.874372: predicting tseg-s0370 
+2024-09-11 15:38:39.518934: tseg-s0370, shape torch.Size([1, 437, 202, 437]), rank 0 
+2024-09-11 15:38:57.234018: predicting tseg-s0385 
+2024-09-11 15:38:57.424066: tseg-s0385, shape torch.Size([1, 182, 242, 242]), rank 0 
+2024-09-11 15:39:01.586006: predicting tseg-s0395 
+2024-09-11 15:39:01.656881: tseg-s0395, shape torch.Size([1, 38, 333, 333]), rank 0 
+2024-09-11 15:39:06.999526: predicting tseg-s0413 
+2024-09-11 15:39:07.291722: tseg-s0413, shape torch.Size([1, 204, 261, 261]), rank 0 
+2024-09-11 15:39:13.121565: predicting tseg-s0421 
+2024-09-11 15:39:13.311814: tseg-s0421, shape torch.Size([1, 171, 244, 244]), rank 0 
+2024-09-11 15:39:17.476803: predicting tseg-s0452 
+2024-09-11 15:39:17.658292: tseg-s0452, shape torch.Size([1, 182, 211, 211]), rank 0 
+2024-09-11 15:39:20.566069: predicting tseg-s0454 
+2024-09-11 15:39:20.940674: tseg-s0454, shape torch.Size([1, 221, 311, 311]), rank 0 
+2024-09-11 15:39:31.508420: predicting tseg-s0455 
+2024-09-11 15:39:31.726700: tseg-s0455, shape torch.Size([1, 207, 219, 219]), rank 0 
+2024-09-11 15:39:36.443608: predicting tseg-s0456 
+2024-09-11 15:39:36.882854: tseg-s0456, shape torch.Size([1, 390, 265, 265]), rank 0 
+2024-09-11 15:39:46.355769: predicting tseg-s0460 
+2024-09-11 15:39:46.398904: tseg-s0460, shape torch.Size([1, 102, 139, 114]), rank 0 
+2024-09-11 15:39:48.424260: predicting tseg-s0475 
+2024-09-11 15:39:48.672071: tseg-s0475, shape torch.Size([1, 225, 277, 277]), rank 0 
+2024-09-11 15:39:53.224854: predicting tseg-s0476 
+2024-09-11 15:39:53.883873: tseg-s0476, shape torch.Size([1, 463, 283, 283]), rank 0 
+2024-09-11 15:40:03.805820: predicting tseg-s0480 
+2024-09-11 15:40:04.242305: tseg-s0480, shape torch.Size([1, 328, 233, 233]), rank 0 
+2024-09-11 15:40:12.395459: predicting tseg-s0490 
+2024-09-11 15:40:12.606292: tseg-s0490, shape torch.Size([1, 193, 237, 237]), rank 0 
+2024-09-11 15:40:18.835870: predicting tseg-s0498 
+2024-09-11 15:40:19.107632: tseg-s0498, shape torch.Size([1, 225, 237, 237]), rank 0 
+2024-09-11 15:40:24.278078: predicting tseg-s0501 
+2024-09-11 15:40:24.307077: tseg-s0501, shape torch.Size([1, 143, 125, 77]), rank 0 
+2024-09-11 15:40:25.267347: predicting tseg-s0508 
+2024-09-11 15:40:25.475317: tseg-s0508, shape torch.Size([1, 219, 221, 221]), rank 0 
+2024-09-11 15:40:30.617367: predicting tseg-s0509 
+2024-09-11 15:40:30.930094: tseg-s0509, shape torch.Size([1, 315, 227, 227]), rank 0 
+2024-09-11 15:40:38.556401: predicting tseg-s0516 
+2024-09-11 15:40:38.999018: tseg-s0516, shape torch.Size([1, 413, 253, 253]), rank 0 
+2024-09-11 15:40:48.885719: predicting tseg-s0521 
+2024-09-11 15:40:48.906779: tseg-s0521, shape torch.Size([1, 133, 120, 53]), rank 0 
+2024-09-11 15:40:50.808405: predicting tseg-s0536 
+2024-09-11 15:40:51.351341: tseg-s0536, shape torch.Size([1, 426, 269, 269]), rank 0 
+2024-09-11 15:41:00.857618: predicting tseg-s0548 
+2024-09-11 15:41:01.200970: tseg-s0548, shape torch.Size([1, 337, 171, 337]), rank 0 
+2024-09-11 15:41:07.484910: predicting tseg-s0573 
+2024-09-11 15:41:07.708768: tseg-s0573, shape torch.Size([1, 247, 223, 223]), rank 0 
+2024-09-11 15:41:12.667338: predicting tseg-s0618 
+2024-09-11 15:41:12.831267: tseg-s0618, shape torch.Size([1, 189, 196, 196]), rank 0 
+2024-09-11 15:41:16.215146: predicting tseg-s0635 
+2024-09-11 15:41:16.722118: tseg-s0635, shape torch.Size([1, 381, 291, 291]), rank 0 
+2024-09-11 15:41:32.065883: predicting tseg-s0636 
+2024-09-11 15:41:32.687350: tseg-s0636, shape torch.Size([1, 477, 277, 277]), rank 0 
+2024-09-11 15:41:43.704839: predicting tseg-s0637 
+2024-09-11 15:41:44.181577: tseg-s0637, shape torch.Size([1, 437, 244, 244]), rank 0 
+2024-09-11 15:41:54.269129: predicting tseg-s0640 
+2024-09-11 15:41:54.359046: tseg-s0640, shape torch.Size([1, 72, 267, 267]), rank 0 
+2024-09-11 15:41:57.401608: predicting tseg-s0642 
+2024-09-11 15:41:57.529014: tseg-s0642, shape torch.Size([1, 99, 143, 143]), rank 0 
+2024-09-11 15:41:58.546741: predicting tseg-s0646 
+2024-09-11 15:41:58.797281: tseg-s0646, shape torch.Size([1, 215, 275, 275]), rank 0 
+2024-09-11 15:42:04.556622: predicting tseg-s0649 
+2024-09-11 15:42:05.019861: tseg-s0649, shape torch.Size([1, 388, 265, 265]), rank 0 
+2024-09-11 15:42:14.644178: predicting tseg-s0650 
+2024-09-11 15:42:15.481563: tseg-s0650, shape torch.Size([1, 462, 333, 333]), rank 0 
+2024-09-11 15:42:35.597201: predicting tseg-s0657 
+2024-09-11 15:42:36.246187: tseg-s0657, shape torch.Size([1, 488, 297, 297]), rank 0 
+2024-09-11 15:43:00.733211: predicting tseg-s0658 
+2024-09-11 15:43:00.963557: tseg-s0658, shape torch.Size([1, 215, 247, 247]), rank 0 
+2024-09-11 15:43:06.221485: predicting tseg-s0660 
+2024-09-11 15:43:06.425958: tseg-s0660, shape torch.Size([1, 252, 202, 202]), rank 0 
+2024-09-11 15:43:11.374661: predicting tseg-s0664 
+2024-09-11 15:43:12.068846: tseg-s0664, shape torch.Size([1, 405, 332, 332]), rank 0 
+2024-09-11 15:43:31.869711: predicting tseg-s0671 
+2024-09-11 15:43:31.946428: tseg-s0671, shape torch.Size([1, 87, 205, 205]), rank 0 
+2024-09-11 15:43:35.494915: predicting tseg-s0672 
+2024-09-11 15:43:35.651243: tseg-s0672, shape torch.Size([1, 200, 200, 200]), rank 0 
+2024-09-11 15:43:41.057059: predicting tseg-s0676 
+2024-09-11 15:43:41.115931: tseg-s0676, shape torch.Size([1, 38, 333, 333]), rank 0 
+2024-09-11 15:43:47.021440: predicting tseg-s0677 
+2024-09-11 15:43:47.252157: tseg-s0677, shape torch.Size([1, 145, 276, 276]), rank 0 
+2024-09-11 15:43:50.228077: predicting tseg-s0681 
+2024-09-11 15:43:50.381910: tseg-s0681, shape torch.Size([1, 167, 203, 203]), rank 0 
+2024-09-11 15:43:53.791317: predicting tseg-s0682 
+2024-09-11 15:43:54.184090: tseg-s0682, shape torch.Size([1, 317, 317, 206]), rank 0 
+2024-09-11 15:44:05.180116: predicting tseg-s0683 
+2024-09-11 15:44:05.675859: tseg-s0683, shape torch.Size([1, 288, 309, 309]), rank 0 
+2024-09-11 15:44:16.999524: predicting tseg-s0690 
+2024-09-11 15:44:17.471372: tseg-s0690, shape torch.Size([1, 538, 162, 281]), rank 0 
+2024-09-11 15:44:25.107952: predicting tseg-s0695 
+2024-09-11 15:44:25.266692: tseg-s0695, shape torch.Size([1, 167, 219, 219]), rank 0 
+2024-09-11 15:44:29.912467: predicting tseg-s0698 
+2024-09-11 15:44:30.148532: tseg-s0698, shape torch.Size([1, 185, 239, 239]), rank 0 
+2024-09-11 15:44:33.251875: predicting tseg-s0699 
+2024-09-11 15:44:33.549918: tseg-s0699, shape torch.Size([1, 258, 248, 248]), rank 0 
+2024-09-11 15:44:38.527636: predicting tseg-s0705 
+2024-09-11 15:44:38.926532: tseg-s0705, shape torch.Size([1, 301, 267, 267]), rank 0 
+2024-09-11 15:44:45.830054: predicting tseg-s0709 
+2024-09-11 15:44:46.081931: tseg-s0709, shape torch.Size([1, 274, 215, 215]), rank 0 
+2024-09-11 15:44:51.454467: predicting tseg-s0715 
+2024-09-11 15:44:51.498870: tseg-s0715, shape torch.Size([1, 47, 225, 225]), rank 0 
+2024-09-11 15:44:54.631855: predicting tseg-s0729 
+2024-09-11 15:44:54.780576: tseg-s0729, shape torch.Size([1, 206, 155, 197]), rank 0 
+2024-09-11 15:44:57.734489: predicting tseg-s0743 
+2024-09-11 15:44:57.955478: tseg-s0743, shape torch.Size([1, 207, 231, 231]), rank 0 
+2024-09-11 15:45:03.205392: predicting tseg-s0756 
+2024-09-11 15:45:03.267617: tseg-s0756, shape torch.Size([1, 182, 129, 129]), rank 0 
+2024-09-11 15:45:04.556480: predicting tseg-s0765 
+2024-09-11 15:45:04.902002: tseg-s0765, shape torch.Size([1, 396, 228, 228]), rank 0 
+2024-09-11 15:45:14.179216: predicting tseg-s0773 
+2024-09-11 15:45:14.347950: tseg-s0773, shape torch.Size([1, 190, 221, 221]), rank 0 
+2024-09-11 15:45:17.364441: predicting tseg-s0782 
+2024-09-11 15:45:17.432534: tseg-s0782, shape torch.Size([1, 79, 177, 255]), rank 0 
+2024-09-11 15:45:20.773692: predicting tseg-s0787 
+2024-09-11 15:45:20.912895: tseg-s0787, shape torch.Size([1, 155, 221, 221]), rank 0 
+2024-09-11 15:45:23.946516: predicting tseg-s0820 
+2024-09-11 15:45:24.134807: tseg-s0820, shape torch.Size([1, 252, 193, 193]), rank 0 
+2024-09-11 15:45:30.053610: predicting tseg-s0839 
+2024-09-11 15:45:30.077007: tseg-s0839, shape torch.Size([1, 83, 96, 96]), rank 0 
+2024-09-11 15:45:31.889846: predicting tseg-s0842 
+2024-09-11 15:45:32.316487: tseg-s0842, shape torch.Size([1, 450, 244, 244]), rank 0 
+2024-09-11 15:45:41.883389: predicting tseg-s0850 
+2024-09-11 15:45:41.975946: tseg-s0850, shape torch.Size([1, 275, 133, 133]), rank 0 
+2024-09-11 15:45:44.613921: predicting tseg-s0852 
+2024-09-11 15:45:44.674370: tseg-s0852, shape torch.Size([1, 37, 219, 219]), rank 0 
+2024-09-11 15:45:47.802595: predicting tseg-s0859 
+2024-09-11 15:45:48.208175: tseg-s0859, shape torch.Size([1, 450, 240, 240]), rank 0 
+2024-09-11 15:45:58.265607: predicting tseg-s0878 
+2024-09-11 15:45:58.771012: tseg-s0878, shape torch.Size([1, 427, 274, 274]), rank 0 
+2024-09-11 15:46:08.581841: predicting tseg-s0898 
+2024-09-11 15:46:08.767381: tseg-s0898, shape torch.Size([1, 185, 233, 233]), rank 0 
+2024-09-11 15:46:12.051209: predicting tseg-s0919 
+2024-09-11 15:46:12.330306: tseg-s0919, shape torch.Size([1, 415, 168, 221]), rank 0 
+2024-09-11 15:46:18.935464: predicting tseg-s0921 
+2024-09-11 15:46:19.040508: tseg-s0921, shape torch.Size([1, 94, 259, 259]), rank 0 
+2024-09-11 15:46:21.925604: predicting tseg-s0922 
+2024-09-11 15:46:21.973984: tseg-s0922, shape torch.Size([1, 120, 143, 120]), rank 0 
+2024-09-11 15:46:24.026375: predicting tseg-s0934 
+2024-09-11 15:46:24.443188: tseg-s0934, shape torch.Size([1, 293, 293, 293]), rank 0 
+2024-09-11 15:46:39.997542: predicting tseg-s0938 
+2024-09-11 15:46:40.067672: tseg-s0938, shape torch.Size([1, 70, 238, 238]), rank 0 
+2024-09-11 15:46:43.724188: predicting tseg-s0947 
+2024-09-11 15:46:43.904665: tseg-s0947, shape torch.Size([1, 167, 207, 207]), rank 0 
+2024-09-11 15:46:47.043308: predicting tseg-s0952 
+2024-09-11 15:46:47.143323: tseg-s0952, shape torch.Size([1, 269, 118, 118]), rank 0 
+2024-09-11 15:46:49.835108: predicting tseg-s0958 
+2024-09-11 15:46:49.913436: tseg-s0958, shape torch.Size([1, 131, 151, 151]), rank 0 
+2024-09-11 15:46:51.595983: predicting tseg-s0959 
+2024-09-11 15:46:51.979954: tseg-s0959, shape torch.Size([1, 413, 233, 233]), rank 0 
+2024-09-11 15:47:02.038716: predicting tseg-s0961 
+2024-09-11 15:47:02.492978: tseg-s0961, shape torch.Size([1, 413, 253, 253]), rank 0 
+2024-09-11 15:47:12.315862: predicting tseg-s0971 
+2024-09-11 15:47:12.550562: tseg-s0971, shape torch.Size([1, 183, 285, 285]), rank 0 
+2024-09-11 15:47:16.689226: predicting tseg-s0972 
+2024-09-11 15:47:16.741230: tseg-s0972, shape torch.Size([1, 128, 142, 142]), rank 0 
+2024-09-11 15:47:18.857023: predicting tseg-s0973 
+2024-09-11 15:47:18.995258: tseg-s0973, shape torch.Size([1, 200, 143, 205]), rank 0 
+2024-09-11 15:47:22.063305: predicting tseg-s0987 
+2024-09-11 15:47:22.103767: tseg-s0987, shape torch.Size([1, 134, 116, 116]), rank 0 
+2024-09-11 15:47:23.247091: predicting tseg-s0991 
+2024-09-11 15:47:23.566370: tseg-s0991, shape torch.Size([1, 411, 208, 208]), rank 0 
+2024-09-11 15:47:32.910937: predicting tseg-s0996 
+2024-09-11 15:47:32.977835: tseg-s0996, shape torch.Size([1, 64, 242, 242]), rank 0 
+2024-09-11 15:47:36.201883: predicting tseg-s1001 
+2024-09-11 15:47:36.323148: tseg-s1001, shape torch.Size([1, 159, 173, 173]), rank 0 
+2024-09-11 15:47:38.609835: predicting tseg-s1006 
+2024-09-11 15:47:39.116134: tseg-s1006, shape torch.Size([1, 415, 264, 264]), rank 0 
+2024-09-11 15:47:48.795296: predicting tseg-s1016 
+2024-09-11 15:47:49.651943: tseg-s1016, shape torch.Size([1, 471, 333, 333]), rank 0 
+2024-09-11 15:48:09.846838: predicting tseg-s1017 
+2024-09-11 15:48:09.960303: tseg-s1017, shape torch.Size([1, 277, 144, 144]), rank 0 
+2024-09-11 15:48:12.048384: predicting tseg-s1024 
+2024-09-11 15:48:12.841198: tseg-s1024, shape torch.Size([1, 415, 333, 333]), rank 0 
+2024-09-11 15:48:33.487054: predicting tseg-s1025 
+2024-09-11 15:48:33.528234: tseg-s1025, shape torch.Size([1, 123, 118, 118]), rank 0 
+2024-09-11 15:48:35.630943: predicting tseg-s1026 
+2024-09-11 15:48:35.715674: tseg-s1026, shape torch.Size([1, 147, 145, 145]), rank 0 
+2024-09-11 15:48:37.211678: predicting tseg-s1031 
+2024-09-11 15:48:37.781932: tseg-s1031, shape torch.Size([1, 419, 287, 287]), rank 0 
+2024-09-11 15:48:47.114262: predicting tseg-s1044 
+2024-09-11 15:48:47.652913: tseg-s1044, shape torch.Size([1, 459, 257, 257]), rank 0 
+2024-09-11 15:48:56.827708: predicting tseg-s1048 
+2024-09-11 15:48:56.865227: tseg-s1048, shape torch.Size([1, 33, 235, 235]), rank 0 
+2024-09-11 15:48:59.508470: predicting tseg-s1049 
+2024-09-11 15:48:59.686770: tseg-s1049, shape torch.Size([1, 133, 221, 221]), rank 0 
+2024-09-11 15:49:02.409835: predicting tseg-s1052 
+2024-09-11 15:49:02.634907: tseg-s1052, shape torch.Size([1, 221, 236, 236]), rank 0 
+2024-09-11 15:49:07.366782: predicting tseg-s1066 
+2024-09-11 15:49:07.579480: tseg-s1066, shape torch.Size([1, 203, 235, 235]), rank 0 
+2024-09-11 15:49:12.702661: predicting tseg-s1072 
+2024-09-11 15:49:12.755720: tseg-s1072, shape torch.Size([1, 109, 167, 167]), rank 0 
+2024-09-11 15:49:13.844671: predicting tseg-s1093 
+2024-09-11 15:49:13.991130: tseg-s1093, shape torch.Size([1, 239, 199, 199]), rank 0 
+2024-09-11 15:49:18.776337: predicting tseg-s1097 
+2024-09-11 15:49:18.850184: tseg-s1097, shape torch.Size([1, 127, 108, 108]), rank 0 
+2024-09-11 15:49:20.332702: predicting tseg-s1105 
+2024-09-11 15:49:20.685963: tseg-s1105, shape torch.Size([1, 407, 221, 221]), rank 0 
+2024-09-11 15:49:30.600305: predicting tseg-s1120 
+2024-09-11 15:49:31.056652: tseg-s1120, shape torch.Size([1, 407, 247, 247]), rank 0 
+2024-09-11 15:49:41.473990: predicting tseg-s1140 
+2024-09-11 15:49:41.836769: tseg-s1140, shape torch.Size([1, 233, 253, 253]), rank 0 
+2024-09-11 15:49:46.522774: predicting tseg-s1144 
+2024-09-11 15:49:46.754643: tseg-s1144, shape torch.Size([1, 225, 228, 228]), rank 0 
+2024-09-11 15:49:51.769601: predicting tseg-s1149 
+2024-09-11 15:49:52.004099: tseg-s1149, shape torch.Size([1, 329, 183, 215]), rank 0 
+2024-09-11 15:49:56.288889: predicting tseg-s1150 
+2024-09-11 15:49:56.334206: tseg-s1150, shape torch.Size([1, 101, 167, 167]), rank 0 
+2024-09-11 15:49:57.825839: predicting tseg-s1179 
+2024-09-11 15:49:57.883091: tseg-s1179, shape torch.Size([1, 136, 117, 117]), rank 0 
+2024-09-11 15:49:58.901908: predicting tseg-s1185 
+2024-09-11 15:49:58.950892: tseg-s1185, shape torch.Size([1, 172, 106, 133]), rank 0 
+2024-09-11 15:49:59.956174: predicting tseg-s1196 
+2024-09-11 15:50:00.035324: tseg-s1196, shape torch.Size([1, 73, 244, 244]), rank 0 
+2024-09-11 15:50:03.014206: predicting tseg-s1222 
+2024-09-11 15:50:03.091376: tseg-s1222, shape torch.Size([1, 61, 264, 264]), rank 0 
+2024-09-11 15:50:05.835126: predicting tseg-s1224 
+2024-09-11 15:50:06.371402: tseg-s1224, shape torch.Size([1, 452, 272, 272]), rank 0 
+2024-09-11 15:50:15.491425: predicting tseg-s1241 
+2024-09-11 15:50:15.806377: tseg-s1241, shape torch.Size([1, 163, 317, 317]), rank 0 
+2024-09-11 15:50:22.034570: predicting tseg-s1245 
+2024-09-11 15:50:22.217099: tseg-s1245, shape torch.Size([1, 181, 228, 228]), rank 0 
+2024-09-11 15:50:25.602744: predicting tseg-s1252 
+2024-09-11 15:50:25.926575: tseg-s1252, shape torch.Size([1, 506, 120, 276]), rank 0 
+2024-09-11 15:50:32.545990: predicting tseg-s1269 
+2024-09-11 15:50:32.724561: tseg-s1269, shape torch.Size([1, 191, 221, 221]), rank 0 
+2024-09-11 15:50:36.258348: predicting tseg-s1278 
+2024-09-11 15:50:36.665212: tseg-s1278, shape torch.Size([1, 239, 304, 304]), rank 0 
+2024-09-11 15:50:47.281194: predicting tseg-s1280 
+2024-09-11 15:50:47.408512: tseg-s1280, shape torch.Size([1, 144, 221, 221]), rank 0 
+2024-09-11 15:50:51.389398: predicting tseg-s1281 
+2024-09-11 15:50:51.629098: tseg-s1281, shape torch.Size([1, 217, 211, 211]), rank 0 
+2024-09-11 15:50:56.928339: predicting tseg-s1287 
+2024-09-11 15:50:57.327010: tseg-s1287, shape torch.Size([1, 440, 239, 239]), rank 0 
+2024-09-11 15:51:06.657785: predicting tseg-s1288 
+2024-09-11 15:51:06.824239: tseg-s1288, shape torch.Size([1, 174, 249, 249]), rank 0 
+2024-09-11 15:51:10.579380: predicting tseg-s1291 
+2024-09-11 15:51:10.887608: tseg-s1291, shape torch.Size([1, 314, 244, 244]), rank 0 
+2024-09-11 15:51:18.716779: predicting tseg-s1303 
+2024-09-11 15:51:18.824845: tseg-s1303, shape torch.Size([1, 387, 113, 113]), rank 0 
+2024-09-11 15:51:23.220937: predicting tseg-s1314 
+2024-09-11 15:51:24.129713: tseg-s1314, shape torch.Size([1, 546, 333, 333]), rank 0 
+2024-09-11 15:51:50.177865: predicting tseg-s1321 
+2024-09-11 15:51:50.826770: tseg-s1321, shape torch.Size([1, 540, 275, 275]), rank 0 
+2024-09-11 15:52:03.099888: predicting tseg-s1325 
+2024-09-11 15:52:03.123472: tseg-s1325, shape torch.Size([1, 77, 117, 124]), rank 0 
+2024-09-11 15:52:04.483666: predicting tseg-s1330 
+2024-09-11 15:52:04.725744: tseg-s1330, shape torch.Size([1, 231, 221, 221]), rank 0 
+2024-09-11 15:52:10.342980: predicting tseg-s1335 
+2024-09-11 15:52:10.384854: tseg-s1335, shape torch.Size([1, 123, 124, 124]), rank 0 
+2024-09-11 15:52:12.419085: predicting tseg-s1337 
+2024-09-11 15:52:12.734698: tseg-s1337, shape torch.Size([1, 231, 280, 280]), rank 0 
+2024-09-11 15:52:18.676626: predicting tseg-s1348 
+2024-09-11 15:52:19.158483: tseg-s1348, shape torch.Size([1, 437, 239, 239]), rank 0 
+2024-09-11 15:52:28.368381: predicting tseg-s1363 
+2024-09-11 15:52:28.941036: tseg-s1363, shape torch.Size([1, 389, 296, 296]), rank 0 
+2024-09-11 15:52:48.597420: predicting tseg-s1365 
+2024-09-11 15:52:49.145975: tseg-s1365, shape torch.Size([1, 438, 277, 277]), rank 0 
+2024-09-11 15:52:59.514684: predicting tseg-s1373 
+2024-09-11 15:52:59.815665: tseg-s1373, shape torch.Size([1, 299, 237, 237]), rank 0 
+2024-09-11 15:53:07.391535: predicting tseg-s1380 
+2024-09-11 15:53:08.169013: tseg-s1380, shape torch.Size([1, 543, 320, 320]), rank 0 
+2024-09-11 15:53:33.103341: predicting tseg-s1388 
+2024-09-11 15:53:34.031970: tseg-s1388, shape torch.Size([1, 525, 316, 316]), rank 0 
+2024-09-11 15:53:58.867141: predicting tseg-s1391 
+2024-09-11 15:53:59.300273: tseg-s1391, shape torch.Size([1, 335, 270, 270]), rank 0 
+2024-09-11 15:55:30.107743: Validation complete 
+2024-09-11 15:55:30.108363: Mean Validation Dice:  0.95026322089997 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_4_16_32_30.txt

@@ -0,0 +1,421 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-04 16:32:30.218034: do_dummy_2d_data_aug: False 
+2024-09-04 16:32:30.311831: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-04 16:32:30.335496: The split file contains 5 splits. 
+2024-09-04 16:32:30.336184: Desired fold for training: 0 
+2024-09-04 16:32:30.336764: This split has 1324 training and 332 validation cases. 
+2024-09-04 16:32:47.851581: Using torch.compile... 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-04 16:32:51.586586: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-04 16:32:52.534976:  
+2024-09-04 16:32:52.546859: Epoch 0 
+2024-09-04 16:32:52.550118: Current learning rate: 0.01 
+2024-09-04 16:51:24.612396: train_loss 0.7547 
+2024-09-04 16:51:24.643302: val_loss 0.4523 
+2024-09-04 16:51:24.752502: Pseudo dice [0.5423, 0.6452, 0.54, 0.0, 0.0, 0.0002, 0.7808] 
+2024-09-04 16:51:24.786732: Epoch time: 1112.08 s 
+2024-09-04 16:51:24.865516: Yayy! New best EMA pseudo Dice: 0.3584 
+2024-09-04 16:51:32.985522:  
+2024-09-04 16:51:32.986421: Epoch 1 
+2024-09-04 16:51:32.987107: Current learning rate: 0.00999 
+2024-09-04 16:56:24.137516: train_loss 0.2939 
+2024-09-04 16:56:24.151670: val_loss 0.1497 
+2024-09-04 16:56:24.163294: Pseudo dice [0.7175, 0.7252, 0.7676, 0.4352, 0.0, 0.0, 0.8717] 
+2024-09-04 16:56:24.165364: Epoch time: 291.16 s 
+2024-09-04 16:56:24.168368: Yayy! New best EMA pseudo Dice: 0.3728 
+2024-09-04 16:56:36.555887:  
+2024-09-04 16:56:36.559146: Epoch 2 
+2024-09-04 16:56:36.559916: Current learning rate: 0.00998 
+2024-09-04 17:01:19.423206: train_loss 0.063 
+2024-09-04 17:01:19.442916: val_loss -0.0934 
+2024-09-04 17:01:19.444020: Pseudo dice [0.7779, 0.7838, 0.8483, 0.68, 0.0, 0.3531, 0.9309] 
+2024-09-04 17:01:19.445312: Epoch time: 282.87 s 
+2024-09-04 17:01:19.448059: Yayy! New best EMA pseudo Dice: 0.398 
+2024-09-04 17:01:27.934186:  
+2024-09-04 17:01:27.935896: Epoch 3 
+2024-09-04 17:01:27.937940: Current learning rate: 0.00997 
+2024-09-04 17:06:03.374408: train_loss -0.1004 
+2024-09-04 17:06:03.381803: val_loss -0.2345 
+2024-09-04 17:06:03.382577: Pseudo dice [0.8125, 0.7954, 0.8747, 0.7598, 0.0003, 0.5964, 0.9404] 
+2024-09-04 17:06:03.383222: Epoch time: 275.44 s 
+2024-09-04 17:06:03.384866: Yayy! New best EMA pseudo Dice: 0.4265 
+2024-09-04 17:06:15.923822:  
+2024-09-04 17:06:15.924788: Epoch 4 
+2024-09-04 17:06:15.925978: Current learning rate: 0.00996 
+2024-09-04 17:10:45.592276: train_loss -0.2565 
+2024-09-04 17:10:45.616945: val_loss -0.3592 
+2024-09-04 17:10:45.617808: Pseudo dice [0.8331, 0.8255, 0.9009, 0.8142, 0.5395, 0.7342, 0.9608] 
+2024-09-04 17:10:45.618937: Epoch time: 269.67 s 
+2024-09-04 17:10:45.619994: Yayy! New best EMA pseudo Dice: 0.4639 
+2024-09-04 17:10:53.790197:  
+2024-09-04 17:10:53.791399: Epoch 5 
+2024-09-04 17:10:53.792274: Current learning rate: 0.00995 
+2024-09-04 17:15:14.032312: train_loss -0.3292 
+2024-09-04 17:15:14.043661: val_loss -0.3818 
+2024-09-04 17:15:14.044614: Pseudo dice [0.8606, 0.8479, 0.9059, 0.8215, 0.5992, 0.7641, 0.9523] 
+2024-09-04 17:15:14.045571: Epoch time: 260.25 s 
+2024-09-04 17:15:14.046739: Yayy! New best EMA pseudo Dice: 0.4997 
+2024-09-04 17:15:22.602915:  
+2024-09-04 17:15:22.603893: Epoch 6 
+2024-09-04 17:15:22.604948: Current learning rate: 0.00995 
+2024-09-04 17:19:45.656605: train_loss -0.4039 
+2024-09-04 17:19:45.673092: val_loss -0.4176 
+2024-09-04 17:19:45.674106: Pseudo dice [0.8648, 0.8614, 0.916, 0.8345, 0.654, 0.7925, 0.9538] 
+2024-09-04 17:19:45.675675: Epoch time: 263.06 s 
+2024-09-04 17:19:45.676730: Yayy! New best EMA pseudo Dice: 0.5337 
+2024-09-04 17:19:54.990601:  
+2024-09-04 17:19:54.991739: Epoch 7 
+2024-09-04 17:19:54.993209: Current learning rate: 0.00994 
+2024-09-04 17:24:32.373308: train_loss -0.4445 
+2024-09-04 17:24:32.408194: val_loss -0.4712 
+2024-09-04 17:24:32.409995: Pseudo dice [0.8795, 0.8752, 0.9191, 0.8513, 0.6788, 0.8166, 0.9689] 
+2024-09-04 17:24:32.411205: Epoch time: 277.39 s 
+2024-09-04 17:24:32.412353: Yayy! New best EMA pseudo Dice: 0.5659 
+2024-09-04 17:24:41.460957:  
+2024-09-04 17:24:41.462443: Epoch 8 
+2024-09-04 17:24:41.463712: Current learning rate: 0.00993 
+2024-09-04 17:29:26.742865: train_loss -0.4432 
+2024-09-04 17:29:26.754167: val_loss -0.5133 
+2024-09-04 17:29:26.755791: Pseudo dice [0.8897, 0.8842, 0.9284, 0.8782, 0.7271, 0.8676, 0.9682] 
+2024-09-04 17:29:26.757086: Epoch time: 285.29 s 
+2024-09-04 17:29:26.758361: Yayy! New best EMA pseudo Dice: 0.5971 
+2024-09-04 17:29:35.991218:  
+2024-09-04 17:29:35.992455: Epoch 9 
+2024-09-04 17:29:35.993681: Current learning rate: 0.00992 
+2024-09-04 17:34:29.842972: train_loss -0.4569 
+2024-09-04 17:34:29.862843: val_loss -0.4729 
+2024-09-04 17:34:29.865907: Pseudo dice [0.8754, 0.8861, 0.9128, 0.8687, 0.7067, 0.8688, 0.9596] 
+2024-09-04 17:34:29.869218: Epoch time: 293.86 s 
+2024-09-04 17:34:29.872642: Yayy! New best EMA pseudo Dice: 0.6242 
+2024-09-04 17:34:41.889093:  
+2024-09-04 17:34:41.891325: Epoch 10 
+2024-09-04 17:34:41.892658: Current learning rate: 0.00991 
+2024-09-04 17:39:24.977340: train_loss -0.5082 
+2024-09-04 17:39:25.006922: val_loss -0.5397 
+2024-09-04 17:39:25.008078: Pseudo dice [0.9059, 0.8942, 0.9358, 0.8808, 0.7215, 0.8668, 0.9758] 
+2024-09-04 17:39:25.009228: Epoch time: 283.09 s 
+2024-09-04 17:39:25.009968: Yayy! New best EMA pseudo Dice: 0.6501 
+2024-09-04 17:39:33.939900:  
+2024-09-04 17:39:33.940859: Epoch 11 
+2024-09-04 17:39:33.941608: Current learning rate: 0.0099 
+2024-09-04 17:44:18.562391: train_loss -0.5233 
+2024-09-04 17:44:18.577966: val_loss -0.5443 
+2024-09-04 17:44:18.579406: Pseudo dice [0.9078, 0.9046, 0.9439, 0.8875, 0.7472, 0.8605, 0.9742] 
+2024-09-04 17:44:18.581187: Epoch time: 284.63 s 
+2024-09-04 17:44:18.582305: Yayy! New best EMA pseudo Dice: 0.674 
+2024-09-04 17:44:29.195353:  
+2024-09-04 17:44:29.196336: Epoch 12 
+2024-09-04 17:44:29.197275: Current learning rate: 0.00989 
+2024-09-04 17:49:18.455374: train_loss -0.5204 
+2024-09-04 17:49:18.457193: val_loss -0.5423 
+2024-09-04 17:49:18.458745: Pseudo dice [0.9071, 0.9057, 0.9384, 0.8706, 0.7333, 0.8945, 0.9734] 
+2024-09-04 17:49:18.461026: Epoch time: 289.26 s 
+2024-09-04 17:49:18.461983: Yayy! New best EMA pseudo Dice: 0.6955 
+2024-09-04 17:49:28.664373:  
+2024-09-04 17:49:28.665843: Epoch 13 
+2024-09-04 17:49:28.667287: Current learning rate: 0.00988 
+2024-09-04 17:54:18.317021: train_loss -0.523 
+2024-09-04 17:54:18.331530: val_loss -0.5482 
+2024-09-04 17:54:18.333296: Pseudo dice [0.8981, 0.9085, 0.9331, 0.8746, 0.7353, 0.8892, 0.9675] 
+2024-09-04 17:54:18.335130: Epoch time: 289.66 s 
+2024-09-04 17:54:18.336870: Yayy! New best EMA pseudo Dice: 0.7146 
+2024-09-04 17:54:26.259611:  
+2024-09-04 17:54:26.260521: Epoch 14 
+2024-09-04 17:54:26.261147: Current learning rate: 0.00987 
+2024-09-04 17:58:59.624392: train_loss -0.5438 
+2024-09-04 17:58:59.652597: val_loss -0.5228 
+2024-09-04 17:58:59.653479: Pseudo dice [0.9045, 0.8953, 0.9289, 0.8765, 0.7409, 0.8705, 0.9777] 
+2024-09-04 17:58:59.654440: Epoch time: 273.37 s 
+2024-09-04 17:58:59.655292: Yayy! New best EMA pseudo Dice: 0.7316 
+2024-09-04 17:59:08.628466:  
+2024-09-04 17:59:08.629604: Epoch 15 
+2024-09-04 17:59:08.631393: Current learning rate: 0.00986 
+2024-09-04 18:03:43.788582: train_loss -0.5556 
+2024-09-04 18:03:43.790476: val_loss -0.5822 
+2024-09-04 18:03:43.791738: Pseudo dice [0.9193, 0.915, 0.951, 0.8882, 0.7388, 0.8935, 0.9792] 
+2024-09-04 18:03:43.792913: Epoch time: 275.17 s 
+2024-09-04 18:03:43.794135: Yayy! New best EMA pseudo Dice: 0.7483 
+2024-09-04 18:03:53.143680:  
+2024-09-04 18:03:53.144819: Epoch 16 
+2024-09-04 18:03:53.145871: Current learning rate: 0.00986 
+2024-09-04 18:08:36.780902: train_loss -0.5631 
+2024-09-04 18:08:36.783308: val_loss -0.5927 
+2024-09-04 18:08:36.794814: Pseudo dice [0.9193, 0.9048, 0.9518, 0.8734, 0.7429, 0.9019, 0.9648] 
+2024-09-04 18:08:36.796035: Epoch time: 283.64 s 
+2024-09-04 18:08:36.797330: Yayy! New best EMA pseudo Dice: 0.7628 
+2024-09-04 18:08:46.369114:  
+2024-09-04 18:08:46.370061: Epoch 17 
+2024-09-04 18:08:46.370837: Current learning rate: 0.00985 
+2024-09-04 18:13:39.265606: train_loss -0.5752 
+2024-09-04 18:13:39.267560: val_loss -0.5819 
+2024-09-04 18:13:39.268833: Pseudo dice [0.9198, 0.912, 0.9451, 0.8834, 0.7479, 0.9102, 0.9766] 
+2024-09-04 18:13:39.269947: Epoch time: 292.9 s 
+2024-09-04 18:13:39.271144: Yayy! New best EMA pseudo Dice: 0.7765 
+2024-09-04 18:13:49.091163:  
+2024-09-04 18:13:49.092105: Epoch 18 
+2024-09-04 18:13:49.092958: Current learning rate: 0.00984 
+2024-09-04 18:18:24.432969: train_loss -0.57 
+2024-09-04 18:18:24.447181: val_loss -0.6306 
+2024-09-04 18:18:24.448127: Pseudo dice [0.9243, 0.9143, 0.9547, 0.9028, 0.778, 0.9133, 0.9795] 
+2024-09-04 18:18:24.449202: Epoch time: 275.34 s 
+2024-09-04 18:18:24.450162: Yayy! New best EMA pseudo Dice: 0.7898 
+2024-09-04 18:18:36.707651:  
+2024-09-04 18:18:36.708612: Epoch 19 
+2024-09-04 18:18:36.709405: Current learning rate: 0.00983 
+2024-09-04 18:23:21.543234: train_loss -0.5984 
+2024-09-04 18:23:21.550392: val_loss -0.6191 
+2024-09-04 18:23:21.551593: Pseudo dice [0.9317, 0.9244, 0.96, 0.9122, 0.7736, 0.9162, 0.9795] 
+2024-09-04 18:23:21.552891: Epoch time: 284.84 s 
+2024-09-04 18:23:21.553576: Yayy! New best EMA pseudo Dice: 0.8022 
+2024-09-04 18:23:30.312485:  
+2024-09-04 18:23:30.313557: Epoch 20 
+2024-09-04 18:23:30.314290: Current learning rate: 0.00982 
+2024-09-04 18:28:14.552891: train_loss -0.5788 
+2024-09-04 18:28:14.569499: val_loss -0.6177 
+2024-09-04 18:28:14.570488: Pseudo dice [0.9256, 0.9221, 0.95, 0.8957, 0.7734, 0.9128, 0.98] 
+2024-09-04 18:28:14.572204: Epoch time: 284.24 s 
+2024-09-04 18:28:14.572948: Yayy! New best EMA pseudo Dice: 0.8128 
+2024-09-04 18:28:25.412246:  
+2024-09-04 18:28:25.413156: Epoch 21 
+2024-09-04 18:28:25.413829: Current learning rate: 0.00981 
+2024-09-04 18:33:13.218303: train_loss -0.5958 
+2024-09-04 18:33:13.234316: val_loss -0.6365 
+2024-09-04 18:33:13.235354: Pseudo dice [0.9269, 0.9219, 0.9533, 0.8952, 0.7518, 0.9096, 0.9824] 
+2024-09-04 18:33:13.236520: Epoch time: 287.81 s 
+2024-09-04 18:33:13.237374: Yayy! New best EMA pseudo Dice: 0.8221 
+2024-09-04 18:33:23.928073:  
+2024-09-04 18:33:23.929306: Epoch 22 
+2024-09-04 18:33:23.930051: Current learning rate: 0.0098 
+2024-09-04 18:38:15.768845: train_loss -0.5779 
+2024-09-04 18:38:15.778701: val_loss -0.6482 
+2024-09-04 18:38:15.780410: Pseudo dice [0.9302, 0.924, 0.9597, 0.9098, 0.7807, 0.9052, 0.9821] 
+2024-09-04 18:38:15.781348: Epoch time: 291.85 s 
+2024-09-04 18:38:15.782116: Yayy! New best EMA pseudo Dice: 0.8312 
+2024-09-04 18:38:24.616342:  
+2024-09-04 18:38:24.617306: Epoch 23 
+2024-09-04 18:38:24.618434: Current learning rate: 0.00979 
+2024-09-04 18:43:10.864866: train_loss -0.6123 
+2024-09-04 18:43:10.892257: val_loss -0.636 
+2024-09-04 18:43:10.893089: Pseudo dice [0.9292, 0.9198, 0.9525, 0.9027, 0.7913, 0.9237, 0.9847] 
+2024-09-04 18:43:10.893968: Epoch time: 286.25 s 
+2024-09-04 18:43:10.894890: Yayy! New best EMA pseudo Dice: 0.8396 
+2024-09-04 18:43:19.525567:  
+2024-09-04 18:43:19.526874: Epoch 24 
+2024-09-04 18:43:19.527814: Current learning rate: 0.00978 
+2024-09-04 18:48:07.164886: train_loss -0.612 
+2024-09-04 18:48:07.179700: val_loss -0.6436 
+2024-09-04 18:48:07.180490: Pseudo dice [0.9327, 0.9288, 0.9614, 0.9116, 0.7967, 0.9263, 0.9839] 
+2024-09-04 18:48:07.181215: Epoch time: 287.64 s 
+2024-09-04 18:48:07.182050: Yayy! New best EMA pseudo Dice: 0.8477 
+2024-09-04 18:48:16.660843:  
+2024-09-04 18:48:16.661720: Epoch 25 
+2024-09-04 18:48:16.662668: Current learning rate: 0.00977 
+2024-09-04 18:52:46.166545: train_loss -0.6198 
+2024-09-04 18:52:46.172196: val_loss -0.6601 
+2024-09-04 18:52:46.173663: Pseudo dice [0.9393, 0.9284, 0.9643, 0.9117, 0.8057, 0.9302, 0.9807] 
+2024-09-04 18:52:46.174593: Epoch time: 269.51 s 
+2024-09-04 18:52:46.175371: Yayy! New best EMA pseudo Dice: 0.8552 
+2024-09-04 18:52:55.552907:  
+2024-09-04 18:52:55.650879: Epoch 26 
+2024-09-04 18:52:55.662783: Current learning rate: 0.00977 
+2024-09-04 18:57:28.899668: train_loss -0.6334 
+2024-09-04 18:57:28.912791: val_loss -0.6534 
+2024-09-04 18:57:28.915751: Pseudo dice [0.9325, 0.9201, 0.9579, 0.9071, 0.7616, 0.925, 0.9821] 
+2024-09-04 18:57:28.917275: Epoch time: 273.35 s 
+2024-09-04 18:57:28.918379: Yayy! New best EMA pseudo Dice: 0.8609 
+2024-09-04 18:57:39.056529:  
+2024-09-04 18:57:39.057535: Epoch 27 
+2024-09-04 18:57:39.058389: Current learning rate: 0.00976 
+2024-09-04 19:02:33.488252: train_loss -0.6186 
+2024-09-04 19:02:33.492045: val_loss -0.6613 
+2024-09-04 19:02:33.493300: Pseudo dice [0.9354, 0.9303, 0.9602, 0.9113, 0.7742, 0.9271, 0.9818] 
+2024-09-04 19:02:33.494189: Epoch time: 294.44 s 
+2024-09-04 19:02:33.495009: Yayy! New best EMA pseudo Dice: 0.8665 
+2024-09-04 19:02:42.617097:  
+2024-09-04 19:02:42.618124: Epoch 28 
+2024-09-04 19:02:42.618894: Current learning rate: 0.00975 
+2024-09-04 19:07:22.019218: train_loss -0.6283 
+2024-09-04 19:07:22.030541: val_loss -0.6497 
+2024-09-04 19:07:22.031543: Pseudo dice [0.9372, 0.9328, 0.9637, 0.9098, 0.7992, 0.9263, 0.9851] 
+2024-09-04 19:07:22.032308: Epoch time: 279.41 s 
+2024-09-04 19:07:22.032980: Yayy! New best EMA pseudo Dice: 0.8721 
+2024-09-04 19:07:31.192338:  
+2024-09-04 19:07:31.193504: Epoch 29 
+2024-09-04 19:07:31.194317: Current learning rate: 0.00974 
+2024-09-04 19:12:13.368532: train_loss -0.6591 
+2024-09-04 19:12:13.371889: val_loss -0.6697 
+2024-09-04 19:12:13.372864: Pseudo dice [0.9435, 0.9299, 0.9687, 0.9203, 0.7864, 0.9375, 0.9822] 
+2024-09-04 19:12:13.373648: Epoch time: 282.18 s 
+2024-09-04 19:12:13.374867: Yayy! New best EMA pseudo Dice: 0.8773 
+2024-09-04 19:12:22.672293:  
+2024-09-04 19:12:22.673205: Epoch 30 
+2024-09-04 19:12:22.673933: Current learning rate: 0.00973 
+2024-09-04 19:17:12.998903: train_loss -0.6534 
+2024-09-04 19:17:13.016715: val_loss -0.672 
+2024-09-04 19:17:13.018040: Pseudo dice [0.9426, 0.9329, 0.9679, 0.9129, 0.8091, 0.9269, 0.9873] 
+2024-09-04 19:17:13.019034: Epoch time: 290.33 s 
+2024-09-04 19:17:13.020182: Yayy! New best EMA pseudo Dice: 0.8821 
+2024-09-04 19:17:23.694266:  
+2024-09-04 19:17:23.695148: Epoch 31 
+2024-09-04 19:17:23.696340: Current learning rate: 0.00972 
+2024-09-04 19:21:59.641176: train_loss -0.6496 
+2024-09-04 19:21:59.642713: val_loss -0.6605 
+2024-09-04 19:21:59.643871: Pseudo dice [0.9394, 0.927, 0.9645, 0.9158, 0.8142, 0.9352, 0.9852] 
+2024-09-04 19:21:59.645078: Epoch time: 275.95 s 
+2024-09-04 19:21:59.646324: Yayy! New best EMA pseudo Dice: 0.8865 
+2024-09-04 19:22:10.994848:  
+2024-09-04 19:22:10.995768: Epoch 32 
+2024-09-04 19:22:10.996507: Current learning rate: 0.00971 
+2024-09-04 19:27:00.267269: train_loss -0.6246 
+2024-09-04 19:27:00.287695: val_loss -0.6521 
+2024-09-04 19:27:00.289492: Pseudo dice [0.937, 0.9344, 0.9589, 0.9148, 0.8121, 0.9225, 0.9864] 
+2024-09-04 19:27:00.291428: Epoch time: 289.28 s 
+2024-09-04 19:27:00.293444: Yayy! New best EMA pseudo Dice: 0.8902 
+2024-09-04 19:27:09.814674:  
+2024-09-04 19:27:09.815859: Epoch 33 
+2024-09-04 19:27:09.817088: Current learning rate: 0.0097 
+2024-09-04 19:32:01.618886: train_loss -0.645 
+2024-09-04 19:32:01.639531: val_loss -0.6831 
+2024-09-04 19:32:01.641026: Pseudo dice [0.9404, 0.9326, 0.9624, 0.9183, 0.8226, 0.9287, 0.9843] 
+2024-09-04 19:32:01.642243: Epoch time: 291.81 s 
+2024-09-04 19:32:01.643702: Yayy! New best EMA pseudo Dice: 0.8939 
+2024-09-04 19:32:11.534695:  
+2024-09-04 19:32:11.546265: Epoch 34 
+2024-09-04 19:32:11.612686: Current learning rate: 0.00969 
+2024-09-04 19:36:52.251600: train_loss -0.6687 
+2024-09-04 19:36:52.260559: val_loss -0.6855 
+2024-09-04 19:36:52.262238: Pseudo dice [0.9431, 0.9322, 0.9622, 0.9128, 0.8006, 0.9408, 0.9857] 
+2024-09-04 19:36:52.264317: Epoch time: 280.72 s 
+2024-09-04 19:36:52.265575: Yayy! New best EMA pseudo Dice: 0.897 
+2024-09-04 19:37:02.565159:  
+2024-09-04 19:37:02.577355: Epoch 35 
+2024-09-04 19:37:02.592764: Current learning rate: 0.00968 
+2024-09-04 19:41:43.568752: train_loss -0.6469 
+2024-09-04 19:41:43.591609: val_loss -0.6886 
+2024-09-04 19:41:43.593227: Pseudo dice [0.9465, 0.9363, 0.9668, 0.9242, 0.7978, 0.9325, 0.9869] 
+2024-09-04 19:41:43.594650: Epoch time: 281.01 s 
+2024-09-04 19:41:43.595665: Yayy! New best EMA pseudo Dice: 0.9001 
+2024-09-04 19:41:57.437129:  
+2024-09-04 19:41:57.438313: Epoch 36 
+2024-09-04 19:41:57.439175: Current learning rate: 0.00968 
+2024-09-04 19:46:30.953845: train_loss -0.658 
+2024-09-04 19:46:31.132960: val_loss -0.6681 
+2024-09-04 19:46:31.150600: Pseudo dice [0.9438, 0.9374, 0.9696, 0.9213, 0.8086, 0.9378, 0.9855] 
+2024-09-04 19:46:31.166784: Epoch time: 273.52 s 
+2024-09-04 19:46:31.178261: Yayy! New best EMA pseudo Dice: 0.903 
+2024-09-04 19:46:40.969545:  
+2024-09-04 19:46:40.970600: Epoch 37 
+2024-09-04 19:46:40.971322: Current learning rate: 0.00967 
+2024-09-04 19:51:10.313670: train_loss -0.6567 
+2024-09-04 19:51:10.326305: val_loss -0.6718 
+2024-09-04 19:51:10.327846: Pseudo dice [0.9435, 0.9346, 0.9603, 0.9194, 0.8131, 0.9425, 0.987] 
+2024-09-04 19:51:10.329134: Epoch time: 269.35 s 
+2024-09-04 19:51:10.330189: Yayy! New best EMA pseudo Dice: 0.9055 
+2024-09-04 19:51:19.291771:  
+2024-09-04 19:51:19.292917: Epoch 38 
+2024-09-04 19:51:19.293958: Current learning rate: 0.00966 
+2024-09-04 19:55:53.196341: train_loss -0.6692 
+2024-09-04 19:55:53.213023: val_loss -0.6345 
+2024-09-04 19:55:53.214654: Pseudo dice [0.9421, 0.9401, 0.9615, 0.9147, 0.8107, 0.94, 0.9825] 
+2024-09-04 19:55:53.215887: Epoch time: 273.91 s 
+2024-09-04 19:55:53.217148: Yayy! New best EMA pseudo Dice: 0.9077 
+2024-09-04 19:56:03.630352:  
+2024-09-04 19:56:03.631492: Epoch 39 
+2024-09-04 19:56:03.632380: Current learning rate: 0.00965 
+2024-09-04 20:00:57.705683: train_loss -0.6787 
+2024-09-04 20:00:57.720947: val_loss -0.7142 
+2024-09-04 20:00:57.722802: Pseudo dice [0.9431, 0.9326, 0.965, 0.9181, 0.813, 0.9403, 0.9856] 
+2024-09-04 20:00:57.724499: Epoch time: 294.08 s 
+2024-09-04 20:00:57.725927: Yayy! New best EMA pseudo Dice: 0.9098 
+2024-09-04 20:01:06.429250:  
+2024-09-04 20:01:06.430412: Epoch 40 
+2024-09-04 20:01:06.431465: Current learning rate: 0.00964 
+2024-09-04 20:05:41.807588: train_loss -0.6756 
+2024-09-04 20:05:41.824279: val_loss -0.6753 
+2024-09-04 20:05:41.825323: Pseudo dice [0.9473, 0.9414, 0.9647, 0.9235, 0.8217, 0.9312, 0.9859] 
+2024-09-04 20:05:41.826500: Epoch time: 275.38 s 
+2024-09-04 20:05:41.827209: Yayy! New best EMA pseudo Dice: 0.9119 
+2024-09-04 20:05:50.679820:  
+2024-09-04 20:05:50.681713: Epoch 41 
+2024-09-04 20:05:50.682781: Current learning rate: 0.00963 
+2024-09-04 20:10:30.328226: train_loss -0.6558 
+2024-09-04 20:10:30.347022: val_loss -0.6969 
+2024-09-04 20:10:30.348571: Pseudo dice [0.9474, 0.9403, 0.9687, 0.9339, 0.8349, 0.9438, 0.9851] 
+2024-09-04 20:10:30.349951: Epoch time: 279.65 s 
+2024-09-04 20:10:30.351240: Yayy! New best EMA pseudo Dice: 0.9143 
+2024-09-04 20:10:42.991168:  
+2024-09-04 20:10:42.992094: Epoch 42 
+2024-09-04 20:10:42.992786: Current learning rate: 0.00962 
+2024-09-04 20:15:20.561590: train_loss -0.6464 
+2024-09-04 20:15:20.578927: val_loss -0.6551 
+2024-09-04 20:15:20.580329: Pseudo dice [0.9368, 0.9298, 0.9601, 0.9165, 0.8036, 0.9412, 0.9844] 
+2024-09-04 20:15:20.581067: Epoch time: 277.57 s 
+2024-09-04 20:15:20.581661: Yayy! New best EMA pseudo Dice: 0.9154 
+2024-09-04 20:15:30.264488:  
+2024-09-04 20:15:30.265436: Epoch 43 
+2024-09-04 20:15:30.266375: Current learning rate: 0.00961 
+2024-09-04 20:19:59.850217: train_loss -0.6108 
+2024-09-04 20:19:59.867406: val_loss -0.644 
+2024-09-04 20:19:59.869079: Pseudo dice [0.9345, 0.9238, 0.9612, 0.9041, 0.7874, 0.9288, 0.979] 
+2024-09-04 20:19:59.870160: Epoch time: 269.59 s 
+2024-09-04 20:19:59.871398: Yayy! New best EMA pseudo Dice: 0.9155 
+2024-09-04 20:20:09.463664:  
+2024-09-04 20:20:09.464597: Epoch 44 
+2024-09-04 20:20:09.465289: Current learning rate: 0.0096 
+2024-09-04 20:24:47.967054: train_loss -0.6369 
+2024-09-04 20:24:47.994912: val_loss -0.6531 
+2024-09-04 20:24:47.995966: Pseudo dice [0.9438, 0.9357, 0.9636, 0.9217, 0.7969, 0.9355, 0.985] 
+2024-09-04 20:24:47.996834: Epoch time: 278.51 s 
+2024-09-04 20:24:47.997881: Yayy! New best EMA pseudo Dice: 0.9166 
+2024-09-04 20:24:57.418353:  
+2024-09-04 20:24:57.419597: Epoch 45 
+2024-09-04 20:24:57.420449: Current learning rate: 0.00959 
+2024-09-04 20:29:42.877529: train_loss -0.6429 
+2024-09-04 20:29:42.880110: val_loss -0.6677 
+2024-09-04 20:29:42.881012: Pseudo dice [0.9402, 0.9271, 0.9629, 0.919, 0.7882, 0.9225, 0.9754] 
+2024-09-04 20:29:42.881983: Epoch time: 285.46 s 
+2024-09-04 20:29:42.882914: Yayy! New best EMA pseudo Dice: 0.9168 
+2024-09-04 20:29:50.832762:  
+2024-09-04 20:29:50.833780: Epoch 46 
+2024-09-04 20:29:50.834540: Current learning rate: 0.00959 
+2024-09-04 20:34:31.621396: train_loss -0.6433 
+2024-09-04 20:34:31.624602: val_loss -0.6411 
+2024-09-04 20:34:31.625569: Pseudo dice [0.938, 0.9324, 0.9626, 0.9087, 0.7978, 0.9268, 0.9802] 
+2024-09-04 20:34:31.626339: Epoch time: 280.79 s 
+2024-09-04 20:34:31.627103: Yayy! New best EMA pseudo Dice: 0.9173 
+2024-09-04 20:34:40.600052:  
+2024-09-04 20:34:40.601156: Epoch 47 
+2024-09-04 20:34:40.601946: Current learning rate: 0.00958 
+2024-09-04 20:39:24.346736: train_loss -0.653 
+2024-09-04 20:39:24.364691: val_loss -0.68 
+2024-09-04 20:39:24.365805: Pseudo dice [0.9456, 0.9405, 0.9653, 0.9092, 0.8135, 0.9398, 0.9864] 
+2024-09-04 20:39:24.366634: Epoch time: 283.75 s 
+2024-09-04 20:39:24.367288: Yayy! New best EMA pseudo Dice: 0.9184 
+2024-09-04 20:39:35.388669:  
+2024-09-04 20:39:35.480217: Epoch 48 
+2024-09-04 20:39:35.488962: Current learning rate: 0.00957 
+2024-09-04 20:43:59.941428: train_loss -0.6675 
+2024-09-04 20:43:59.955547: val_loss -0.6975 
+2024-09-04 20:43:59.956751: Pseudo dice [0.9439, 0.9378, 0.9656, 0.9249, 0.8014, 0.9388, 0.9834] 
+2024-09-04 20:43:59.957748: Epoch time: 264.56 s 
+2024-09-04 20:43:59.959002: Yayy! New best EMA pseudo Dice: 0.9193 
+2024-09-04 20:44:08.939043:  
+2024-09-04 20:44:08.940154: Epoch 49 
+2024-09-04 20:44:08.941007: Current learning rate: 0.00956 
+2024-09-04 20:48:50.915356: train_loss -0.69 
+2024-09-04 20:48:50.919242: val_loss -0.6971 
+2024-09-04 20:48:50.920112: Pseudo dice [0.941, 0.935, 0.9677, 0.9179, 0.8085, 0.9399, 0.9826] 
+2024-09-04 20:48:50.920878: Epoch time: 281.98 s 
+2024-09-04 20:48:57.553629: Yayy! New best EMA pseudo Dice: 0.9202 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_5_00_32_01.txt

@@ -0,0 +1,406 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-05 00:32:04.857353: Using torch.compile... 
+2024-09-05 00:32:14.546425: do_dummy_2d_data_aug: False 
+2024-09-05 00:32:14.552568: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-05 00:32:14.553875: The split file contains 5 splits. 
+2024-09-05 00:32:14.554505: Desired fold for training: 0 
+2024-09-05 00:32:14.555068: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-05 00:32:26.637626: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-05 00:32:29.606219:  
+2024-09-05 00:32:29.607365: Epoch 50 
+2024-09-05 00:32:29.610240: Current learning rate: 0.00955 
+2024-09-05 00:50:44.713544: train_loss -0.666 
+2024-09-05 00:50:44.714832: val_loss -0.6859 
+2024-09-05 00:50:44.715503: Pseudo dice [0.9385, 0.9392, 0.9556, 0.9278, 0.813, 0.9389, 0.9853] 
+2024-09-05 00:50:44.716128: Epoch time: 1095.11 s 
+2024-09-05 00:50:44.716744: Yayy! New best EMA pseudo Dice: 0.921 
+2024-09-05 00:50:58.636887:  
+2024-09-05 00:50:58.705183: Epoch 51 
+2024-09-05 00:50:58.742226: Current learning rate: 0.00954 
+2024-09-05 00:55:35.898707: train_loss -0.6779 
+2024-09-05 00:55:35.900918: val_loss -0.6715 
+2024-09-05 00:55:35.915755: Pseudo dice [0.9464, 0.9374, 0.9657, 0.9222, 0.8219, 0.9491, 0.9875] 
+2024-09-05 00:55:35.917289: Epoch time: 277.26 s 
+2024-09-05 00:55:35.919353: Yayy! New best EMA pseudo Dice: 0.9222 
+2024-09-05 00:55:45.843538:  
+2024-09-05 00:55:45.844574: Epoch 52 
+2024-09-05 00:55:45.845566: Current learning rate: 0.00953 
+2024-09-05 01:00:39.537770: train_loss -0.6716 
+2024-09-05 01:00:39.557428: val_loss -0.702 
+2024-09-05 01:00:39.560374: Pseudo dice [0.9469, 0.9395, 0.9656, 0.9317, 0.8235, 0.9465, 0.9877] 
+2024-09-05 01:00:39.562542: Epoch time: 293.7 s 
+2024-09-05 01:00:39.564588: Yayy! New best EMA pseudo Dice: 0.9234 
+2024-09-05 01:00:50.028852:  
+2024-09-05 01:00:50.171027: Epoch 53 
+2024-09-05 01:00:50.196065: Current learning rate: 0.00952 
+2024-09-05 01:05:50.617430: train_loss -0.6707 
+2024-09-05 01:05:50.618985: val_loss -0.6854 
+2024-09-05 01:05:50.620179: Pseudo dice [0.9456, 0.9432, 0.9637, 0.9184, 0.8167, 0.936, 0.9871] 
+2024-09-05 01:05:50.620906: Epoch time: 300.59 s 
+2024-09-05 01:05:50.621560: Yayy! New best EMA pseudo Dice: 0.9241 
+2024-09-05 01:05:58.238500:  
+2024-09-05 01:05:58.239326: Epoch 54 
+2024-09-05 01:05:58.240119: Current learning rate: 0.00951 
+2024-09-05 01:10:31.552188: train_loss -0.6681 
+2024-09-05 01:10:31.561504: val_loss -0.6754 
+2024-09-05 01:10:31.562711: Pseudo dice [0.9507, 0.9463, 0.9672, 0.9251, 0.8066, 0.9417, 0.9864] 
+2024-09-05 01:10:31.563936: Epoch time: 273.32 s 
+2024-09-05 01:10:31.565495: Yayy! New best EMA pseudo Dice: 0.9249 
+2024-09-05 01:10:40.767559:  
+2024-09-05 01:10:40.768440: Epoch 55 
+2024-09-05 01:10:40.769206: Current learning rate: 0.0095 
+2024-09-05 01:15:14.068273: train_loss -0.6682 
+2024-09-05 01:15:14.083787: val_loss -0.6603 
+2024-09-05 01:15:14.086058: Pseudo dice [0.9437, 0.94, 0.9684, 0.9275, 0.8264, 0.9426, 0.9853] 
+2024-09-05 01:15:14.098573: Epoch time: 273.3 s 
+2024-09-05 01:15:14.099905: Yayy! New best EMA pseudo Dice: 0.9257 
+2024-09-05 01:15:24.243653:  
+2024-09-05 01:15:24.244639: Epoch 56 
+2024-09-05 01:15:24.245347: Current learning rate: 0.00949 
+2024-09-05 01:20:00.112716: train_loss -0.6691 
+2024-09-05 01:20:00.114193: val_loss -0.7002 
+2024-09-05 01:20:00.115168: Pseudo dice [0.9497, 0.9417, 0.9675, 0.9282, 0.8216, 0.9417, 0.9886] 
+2024-09-05 01:20:00.115972: Epoch time: 275.87 s 
+2024-09-05 01:20:00.116610: Yayy! New best EMA pseudo Dice: 0.9266 
+2024-09-05 01:20:09.531249:  
+2024-09-05 01:20:09.532138: Epoch 57 
+2024-09-05 01:20:09.532786: Current learning rate: 0.00949 
+2024-09-05 01:24:54.307930: train_loss -0.6989 
+2024-09-05 01:24:54.323140: val_loss -0.6433 
+2024-09-05 01:24:54.324343: Pseudo dice [0.9454, 0.9427, 0.9647, 0.9341, 0.8303, 0.9425, 0.9893] 
+2024-09-05 01:24:54.325421: Epoch time: 284.78 s 
+2024-09-05 01:24:54.327017: Yayy! New best EMA pseudo Dice: 0.9275 
+2024-09-05 01:25:03.499497:  
+2024-09-05 01:25:03.500589: Epoch 58 
+2024-09-05 01:25:03.501464: Current learning rate: 0.00948 
+2024-09-05 01:29:39.341948: train_loss -0.6755 
+2024-09-05 01:29:39.343663: val_loss -0.7074 
+2024-09-05 01:29:39.344664: Pseudo dice [0.9518, 0.9437, 0.9694, 0.9311, 0.8293, 0.9507, 0.9875] 
+2024-09-05 01:29:39.345448: Epoch time: 275.85 s 
+2024-09-05 01:29:39.346465: Yayy! New best EMA pseudo Dice: 0.9285 
+2024-09-05 01:29:48.165974:  
+2024-09-05 01:29:48.166937: Epoch 59 
+2024-09-05 01:29:48.167663: Current learning rate: 0.00947 
+2024-09-05 01:34:30.861431: train_loss -0.7004 
+2024-09-05 01:34:30.869909: val_loss -0.6664 
+2024-09-05 01:34:30.870995: Pseudo dice [0.9469, 0.9441, 0.9665, 0.9228, 0.8237, 0.9468, 0.9864] 
+2024-09-05 01:34:30.872004: Epoch time: 282.7 s 
+2024-09-05 01:34:30.873252: Yayy! New best EMA pseudo Dice: 0.929 
+2024-09-05 01:34:40.340709:  
+2024-09-05 01:34:40.342631: Epoch 60 
+2024-09-05 01:34:40.343294: Current learning rate: 0.00946 
+2024-09-05 01:39:38.137101: train_loss -0.6829 
+2024-09-05 01:39:38.154015: val_loss -0.703 
+2024-09-05 01:39:38.155312: Pseudo dice [0.9495, 0.9445, 0.9664, 0.9305, 0.8127, 0.9475, 0.9876] 
+2024-09-05 01:39:38.156441: Epoch time: 297.8 s 
+2024-09-05 01:39:38.159112: Yayy! New best EMA pseudo Dice: 0.9295 
+2024-09-05 01:39:48.073462:  
+2024-09-05 01:39:48.074967: Epoch 61 
+2024-09-05 01:39:48.075746: Current learning rate: 0.00945 
+2024-09-05 01:44:48.194967: train_loss -0.6812 
+2024-09-05 01:44:48.212122: val_loss -0.6581 
+2024-09-05 01:44:48.213044: Pseudo dice [0.9519, 0.9464, 0.9693, 0.9229, 0.8236, 0.9459, 0.9859] 
+2024-09-05 01:44:48.213841: Epoch time: 300.13 s 
+2024-09-05 01:44:48.215828: Yayy! New best EMA pseudo Dice: 0.9301 
+2024-09-05 01:44:57.340414:  
+2024-09-05 01:44:57.342596: Epoch 62 
+2024-09-05 01:44:57.343770: Current learning rate: 0.00944 
+2024-09-05 01:49:51.226254: train_loss -0.6688 
+2024-09-05 01:49:51.257847: val_loss -0.7136 
+2024-09-05 01:49:51.259135: Pseudo dice [0.9528, 0.945, 0.9705, 0.936, 0.8306, 0.9494, 0.9877] 
+2024-09-05 01:49:51.260118: Epoch time: 293.89 s 
+2024-09-05 01:49:51.261089: Yayy! New best EMA pseudo Dice: 0.931 
+2024-09-05 01:50:01.983498:  
+2024-09-05 01:50:01.984474: Epoch 63 
+2024-09-05 01:50:01.985412: Current learning rate: 0.00943 
+2024-09-05 01:54:46.456017: train_loss -0.6828 
+2024-09-05 01:54:46.472460: val_loss -0.7092 
+2024-09-05 01:54:46.474447: Pseudo dice [0.9526, 0.9452, 0.9704, 0.9296, 0.8211, 0.9512, 0.9846] 
+2024-09-05 01:54:46.476675: Epoch time: 284.48 s 
+2024-09-05 01:54:46.478270: Yayy! New best EMA pseudo Dice: 0.9315 
+2024-09-05 01:54:55.803722:  
+2024-09-05 01:54:55.804576: Epoch 64 
+2024-09-05 01:54:55.805278: Current learning rate: 0.00942 
+2024-09-05 01:59:30.370342: train_loss -0.694 
+2024-09-05 01:59:30.372179: val_loss -0.6903 
+2024-09-05 01:59:30.373056: Pseudo dice [0.9519, 0.9447, 0.9717, 0.935, 0.8396, 0.954, 0.9891] 
+2024-09-05 01:59:30.374134: Epoch time: 274.57 s 
+2024-09-05 01:59:30.374859: Yayy! New best EMA pseudo Dice: 0.9324 
+2024-09-05 01:59:39.732435:  
+2024-09-05 01:59:39.733352: Epoch 65 
+2024-09-05 01:59:39.734106: Current learning rate: 0.00941 
+2024-09-05 02:04:14.223453: train_loss -0.6871 
+2024-09-05 02:04:14.225106: val_loss -0.7212 
+2024-09-05 02:04:14.226405: Pseudo dice [0.9537, 0.9428, 0.9696, 0.9326, 0.8276, 0.953, 0.9879] 
+2024-09-05 02:04:14.227726: Epoch time: 274.49 s 
+2024-09-05 02:04:14.228932: Yayy! New best EMA pseudo Dice: 0.933 
+2024-09-05 02:04:23.957844:  
+2024-09-05 02:04:23.958938: Epoch 66 
+2024-09-05 02:04:23.960122: Current learning rate: 0.0094 
+2024-09-05 02:09:00.069996: train_loss -0.6895 
+2024-09-05 02:09:00.071325: val_loss -0.6728 
+2024-09-05 02:09:00.072145: Pseudo dice [0.9561, 0.9498, 0.9719, 0.9356, 0.8332, 0.9567, 0.987] 
+2024-09-05 02:09:00.072868: Epoch time: 276.12 s 
+2024-09-05 02:09:00.073477: Yayy! New best EMA pseudo Dice: 0.9339 
+2024-09-05 02:09:09.863219:  
+2024-09-05 02:09:09.921320: Epoch 67 
+2024-09-05 02:09:09.970711: Current learning rate: 0.00939 
+2024-09-05 02:14:06.097743: train_loss -0.6976 
+2024-09-05 02:14:06.099289: val_loss -0.6758 
+2024-09-05 02:14:06.100039: Pseudo dice [0.953, 0.9458, 0.9709, 0.9282, 0.8168, 0.95, 0.9866] 
+2024-09-05 02:14:06.100765: Epoch time: 296.24 s 
+2024-09-05 02:14:06.101425: Yayy! New best EMA pseudo Dice: 0.9341 
+2024-09-05 02:14:14.664114:  
+2024-09-05 02:14:14.664978: Epoch 68 
+2024-09-05 02:14:14.665607: Current learning rate: 0.00939 
+2024-09-05 02:18:59.727147: train_loss -0.6824 
+2024-09-05 02:18:59.739378: val_loss -0.7232 
+2024-09-05 02:18:59.740648: Pseudo dice [0.9518, 0.9446, 0.9724, 0.9341, 0.8304, 0.9534, 0.9877] 
+2024-09-05 02:18:59.742047: Epoch time: 285.07 s 
+2024-09-05 02:18:59.743136: Yayy! New best EMA pseudo Dice: 0.9346 
+2024-09-05 02:19:09.975715:  
+2024-09-05 02:19:09.976790: Epoch 69 
+2024-09-05 02:19:09.977535: Current learning rate: 0.00938 
+2024-09-05 02:23:50.739699: train_loss -0.6774 
+2024-09-05 02:23:50.757589: val_loss -0.6727 
+2024-09-05 02:23:50.758693: Pseudo dice [0.9519, 0.9441, 0.971, 0.9258, 0.8339, 0.939, 0.9875] 
+2024-09-05 02:23:50.759561: Epoch time: 280.77 s 
+2024-09-05 02:23:50.760396: Yayy! New best EMA pseudo Dice: 0.9347 
+2024-09-05 02:24:01.619776:  
+2024-09-05 02:24:01.620881: Epoch 70 
+2024-09-05 02:24:01.621812: Current learning rate: 0.00937 
+2024-09-05 02:28:15.868261: train_loss -0.6846 
+2024-09-05 02:28:15.870803: val_loss -0.684 
+2024-09-05 02:28:15.871849: Pseudo dice [0.9514, 0.9438, 0.9699, 0.9339, 0.8217, 0.9515, 0.9876] 
+2024-09-05 02:28:15.873020: Epoch time: 254.25 s 
+2024-09-05 02:28:15.873666: Yayy! New best EMA pseudo Dice: 0.935 
+2024-09-05 02:28:25.436288:  
+2024-09-05 02:28:25.437185: Epoch 71 
+2024-09-05 02:28:25.437797: Current learning rate: 0.00936 
+2024-09-05 02:33:27.600688: train_loss -0.6957 
+2024-09-05 02:33:27.602425: val_loss -0.7211 
+2024-09-05 02:33:27.603283: Pseudo dice [0.9529, 0.949, 0.9737, 0.9339, 0.8345, 0.9555, 0.9868] 
+2024-09-05 02:33:27.604011: Epoch time: 302.17 s 
+2024-09-05 02:33:27.604931: Yayy! New best EMA pseudo Dice: 0.9356 
+2024-09-05 02:33:35.860476:  
+2024-09-05 02:33:35.861429: Epoch 72 
+2024-09-05 02:33:35.862263: Current learning rate: 0.00935 
+2024-09-05 02:38:11.094638: train_loss -0.6943 
+2024-09-05 02:38:11.107465: val_loss -0.7076 
+2024-09-05 02:38:11.108584: Pseudo dice [0.9555, 0.9484, 0.9717, 0.9375, 0.8381, 0.9556, 0.9876] 
+2024-09-05 02:38:11.109715: Epoch time: 275.24 s 
+2024-09-05 02:38:11.110811: Yayy! New best EMA pseudo Dice: 0.9362 
+2024-09-05 02:38:20.609249:  
+2024-09-05 02:38:20.610256: Epoch 73 
+2024-09-05 02:38:20.610952: Current learning rate: 0.00934 
+2024-09-05 02:43:01.260146: train_loss -0.692 
+2024-09-05 02:43:01.275070: val_loss -0.6878 
+2024-09-05 02:43:01.276119: Pseudo dice [0.9538, 0.9479, 0.9715, 0.9363, 0.8377, 0.9479, 0.988] 
+2024-09-05 02:43:01.277088: Epoch time: 280.66 s 
+2024-09-05 02:43:01.278121: Yayy! New best EMA pseudo Dice: 0.9366 
+2024-09-05 02:43:11.010301:  
+2024-09-05 02:43:11.011320: Epoch 74 
+2024-09-05 02:43:11.011992: Current learning rate: 0.00933 
+2024-09-05 02:48:00.586725: train_loss -0.6863 
+2024-09-05 02:48:00.589230: val_loss -0.6816 
+2024-09-05 02:48:00.590130: Pseudo dice [0.9526, 0.9427, 0.9741, 0.9319, 0.8156, 0.9511, 0.9853] 
+2024-09-05 02:48:00.591296: Epoch time: 289.58 s 
+2024-09-05 02:48:03.360751:  
+2024-09-05 02:48:03.375100: Epoch 75 
+2024-09-05 02:48:03.376758: Current learning rate: 0.00932 
+2024-09-05 02:52:57.283037: train_loss -0.6783 
+2024-09-05 02:52:57.286605: val_loss -0.6897 
+2024-09-05 02:52:57.288078: Pseudo dice [0.9449, 0.9382, 0.9613, 0.9221, 0.8236, 0.9312, 0.9864] 
+2024-09-05 02:52:57.289421: Epoch time: 293.93 s 
+2024-09-05 02:53:00.818273:  
+2024-09-05 02:53:00.819642: Epoch 76 
+2024-09-05 02:53:00.820717: Current learning rate: 0.00931 
+2024-09-05 02:57:42.713946: train_loss -0.6664 
+2024-09-05 02:57:42.715797: val_loss -0.711 
+2024-09-05 02:57:42.716834: Pseudo dice [0.9458, 0.9424, 0.9676, 0.9321, 0.8128, 0.9519, 0.9857] 
+2024-09-05 02:57:42.717802: Epoch time: 281.9 s 
+2024-09-05 02:57:47.059335:  
+2024-09-05 02:57:47.071101: Epoch 77 
+2024-09-05 02:57:47.072948: Current learning rate: 0.0093 
+2024-09-05 03:02:47.575464: train_loss -0.6748 
+2024-09-05 03:02:47.593459: val_loss -0.6337 
+2024-09-05 03:02:47.594798: Pseudo dice [0.9327, 0.929, 0.9505, 0.9097, 0.7938, 0.9444, 0.9827] 
+2024-09-05 03:02:47.595932: Epoch time: 300.52 s 
+2024-09-05 03:02:51.195390:  
+2024-09-05 03:02:51.196410: Epoch 78 
+2024-09-05 03:02:51.197325: Current learning rate: 0.0093 
+2024-09-05 03:07:53.565698: train_loss -0.6467 
+2024-09-05 03:07:53.579485: val_loss -0.7055 
+2024-09-05 03:07:53.582584: Pseudo dice [0.9445, 0.9388, 0.9632, 0.9249, 0.7979, 0.9405, 0.9876] 
+2024-09-05 03:07:53.584043: Epoch time: 302.38 s 
+2024-09-05 03:07:57.714933:  
+2024-09-05 03:07:57.715857: Epoch 79 
+2024-09-05 03:07:57.716509: Current learning rate: 0.00929 
+2024-09-05 03:12:29.396415: train_loss -0.6753 
+2024-09-05 03:12:29.411074: val_loss -0.6847 
+2024-09-05 03:12:29.412162: Pseudo dice [0.9473, 0.9422, 0.9667, 0.9299, 0.8183, 0.948, 0.9852] 
+2024-09-05 03:12:29.413049: Epoch time: 271.69 s 
+2024-09-05 03:12:32.142717:  
+2024-09-05 03:12:32.150262: Epoch 80 
+2024-09-05 03:12:32.151416: Current learning rate: 0.00928 
+2024-09-05 03:17:18.118812: train_loss -0.6766 
+2024-09-05 03:17:18.137106: val_loss -0.6944 
+2024-09-05 03:17:18.138120: Pseudo dice [0.9539, 0.9452, 0.9727, 0.9342, 0.8324, 0.9457, 0.9843] 
+2024-09-05 03:17:18.139318: Epoch time: 285.98 s 
+2024-09-05 03:17:21.106868:  
+2024-09-05 03:17:21.107991: Epoch 81 
+2024-09-05 03:17:21.109013: Current learning rate: 0.00927 
+2024-09-05 03:21:58.229145: train_loss -0.6729 
+2024-09-05 03:21:58.240179: val_loss -0.7244 
+2024-09-05 03:21:58.241088: Pseudo dice [0.9501, 0.9431, 0.9706, 0.9251, 0.8312, 0.9473, 0.9871] 
+2024-09-05 03:21:58.241991: Epoch time: 277.13 s 
+2024-09-05 03:22:00.654082:  
+2024-09-05 03:22:00.655357: Epoch 82 
+2024-09-05 03:22:00.656168: Current learning rate: 0.00926 
+2024-09-05 03:26:20.289703: train_loss -0.683 
+2024-09-05 03:26:20.314905: val_loss -0.685 
+2024-09-05 03:26:20.316253: Pseudo dice [0.9528, 0.9429, 0.9706, 0.9309, 0.8343, 0.9483, 0.9866] 
+2024-09-05 03:26:20.317143: Epoch time: 259.64 s 
+2024-09-05 03:26:22.948915:  
+2024-09-05 03:26:22.950395: Epoch 83 
+2024-09-05 03:26:22.951883: Current learning rate: 0.00925 
+2024-09-05 03:31:00.384805: train_loss -0.6803 
+2024-09-05 03:31:00.400467: val_loss -0.7041 
+2024-09-05 03:31:00.401435: Pseudo dice [0.9481, 0.9444, 0.9667, 0.9239, 0.8219, 0.9503, 0.9848] 
+2024-09-05 03:31:00.402587: Epoch time: 277.44 s 
+2024-09-05 03:31:03.769664:  
+2024-09-05 03:31:03.771199: Epoch 84 
+2024-09-05 03:31:03.771974: Current learning rate: 0.00924 
+2024-09-05 03:36:05.690095: train_loss -0.6821 
+2024-09-05 03:36:05.709452: val_loss -0.6769 
+2024-09-05 03:36:05.710881: Pseudo dice [0.9518, 0.947, 0.9709, 0.9288, 0.8448, 0.954, 0.9892] 
+2024-09-05 03:36:05.712079: Epoch time: 301.93 s 
+2024-09-05 03:36:08.842082:  
+2024-09-05 03:36:08.843840: Epoch 85 
+2024-09-05 03:36:08.845258: Current learning rate: 0.00923 
+2024-09-05 03:40:46.907560: train_loss -0.6716 
+2024-09-05 03:40:46.924983: val_loss -0.7143 
+2024-09-05 03:40:46.926442: Pseudo dice [0.954, 0.9478, 0.9717, 0.9369, 0.826, 0.9393, 0.9869] 
+2024-09-05 03:40:46.927313: Epoch time: 278.07 s 
+2024-09-05 03:40:49.967817:  
+2024-09-05 03:40:49.968958: Epoch 86 
+2024-09-05 03:40:49.969985: Current learning rate: 0.00922 
+2024-09-05 03:45:53.603011: train_loss -0.6847 
+2024-09-05 03:45:53.710146: val_loss -0.698 
+2024-09-05 03:45:53.723418: Pseudo dice [0.9588, 0.9509, 0.9768, 0.9354, 0.8347, 0.9554, 0.9879] 
+2024-09-05 03:45:53.784474: Epoch time: 303.64 s 
+2024-09-05 03:45:56.527849:  
+2024-09-05 03:45:56.628083: Epoch 87 
+2024-09-05 03:45:56.660907: Current learning rate: 0.00921 
+2024-09-05 03:50:38.697884: train_loss -0.6941 
+2024-09-05 03:50:38.715154: val_loss -0.7327 
+2024-09-05 03:50:38.716730: Pseudo dice [0.9542, 0.9465, 0.9726, 0.9395, 0.8426, 0.9551, 0.9886] 
+2024-09-05 03:50:38.717924: Epoch time: 282.17 s 
+2024-09-05 03:50:38.718673: Yayy! New best EMA pseudo Dice: 0.9369 
+2024-09-05 03:50:47.588348:  
+2024-09-05 03:50:47.589488: Epoch 88 
+2024-09-05 03:50:47.590379: Current learning rate: 0.0092 
+2024-09-05 03:55:52.910986: train_loss -0.7017 
+2024-09-05 03:55:52.914770: val_loss -0.7089 
+2024-09-05 03:55:52.915883: Pseudo dice [0.9547, 0.9481, 0.9737, 0.9343, 0.8283, 0.9566, 0.9871] 
+2024-09-05 03:55:52.916693: Epoch time: 305.33 s 
+2024-09-05 03:55:52.917501: Yayy! New best EMA pseudo Dice: 0.9372 
+2024-09-05 03:56:03.132331:  
+2024-09-05 03:56:03.133401: Epoch 89 
+2024-09-05 03:56:03.134263: Current learning rate: 0.0092 
+2024-09-05 04:00:27.409684: train_loss -0.7069 
+2024-09-05 04:00:27.425274: val_loss -0.6943 
+2024-09-05 04:00:27.426237: Pseudo dice [0.9526, 0.9441, 0.9726, 0.9273, 0.8316, 0.9525, 0.9871] 
+2024-09-05 04:00:27.427098: Epoch time: 264.28 s 
+2024-09-05 04:00:27.429356: Yayy! New best EMA pseudo Dice: 0.9373 
+2024-09-05 04:00:35.589062:  
+2024-09-05 04:00:35.590019: Epoch 90 
+2024-09-05 04:00:35.591064: Current learning rate: 0.00919 
+2024-09-05 04:05:12.368295: train_loss -0.676 
+2024-09-05 04:05:12.371636: val_loss -0.7418 
+2024-09-05 04:05:12.372845: Pseudo dice [0.9557, 0.9411, 0.974, 0.9364, 0.8349, 0.9528, 0.9882] 
+2024-09-05 04:05:12.373776: Epoch time: 276.78 s 
+2024-09-05 04:05:12.374369: Yayy! New best EMA pseudo Dice: 0.9377 
+2024-09-05 04:05:20.930224:  
+2024-09-05 04:05:20.931069: Epoch 91 
+2024-09-05 04:05:20.931903: Current learning rate: 0.00918 
+2024-09-05 04:10:05.032981: train_loss -0.7013 
+2024-09-05 04:10:05.035286: val_loss -0.686 
+2024-09-05 04:10:05.036085: Pseudo dice [0.9518, 0.9439, 0.9732, 0.9287, 0.8142, 0.9481, 0.9849] 
+2024-09-05 04:10:05.037136: Epoch time: 284.11 s 
+2024-09-05 04:10:07.741518:  
+2024-09-05 04:10:07.744024: Epoch 92 
+2024-09-05 04:10:07.744800: Current learning rate: 0.00917 
+2024-09-05 04:14:41.820854: train_loss -0.7047 
+2024-09-05 04:14:41.845477: val_loss -0.7377 
+2024-09-05 04:14:41.847953: Pseudo dice [0.9573, 0.9499, 0.9709, 0.9413, 0.8474, 0.9565, 0.989] 
+2024-09-05 04:14:41.848968: Epoch time: 274.09 s 
+2024-09-05 04:14:41.849868: Yayy! New best EMA pseudo Dice: 0.9381 
+2024-09-05 04:14:50.620000:  
+2024-09-05 04:14:50.620878: Epoch 93 
+2024-09-05 04:14:50.621497: Current learning rate: 0.00916 
+2024-09-05 04:19:34.318589: train_loss -0.7058 
+2024-09-05 04:19:34.322588: val_loss -0.685 
+2024-09-05 04:19:34.324006: Pseudo dice [0.9537, 0.9484, 0.9711, 0.933, 0.8434, 0.9488, 0.9861] 
+2024-09-05 04:19:34.325122: Epoch time: 283.7 s 
+2024-09-05 04:19:34.326102: Yayy! New best EMA pseudo Dice: 0.9384 
+2024-09-05 04:19:44.123462:  
+2024-09-05 04:19:44.124618: Epoch 94 
+2024-09-05 04:19:44.125493: Current learning rate: 0.00915 
+2024-09-05 04:24:20.165589: train_loss -0.7079 
+2024-09-05 04:24:20.167353: val_loss -0.6983 
+2024-09-05 04:24:20.168311: Pseudo dice [0.9521, 0.9451, 0.9699, 0.9301, 0.8236, 0.9553, 0.9861] 
+2024-09-05 04:24:20.169137: Epoch time: 276.05 s 
+2024-09-05 04:24:22.585601:  
+2024-09-05 04:24:22.586749: Epoch 95 
+2024-09-05 04:24:22.587588: Current learning rate: 0.00914 
+2024-09-05 04:28:51.029822: train_loss -0.696 
+2024-09-05 04:28:51.052405: val_loss -0.7098 
+2024-09-05 04:28:51.053502: Pseudo dice [0.9555, 0.9484, 0.9709, 0.936, 0.8447, 0.9571, 0.9875] 
+2024-09-05 04:28:51.054587: Epoch time: 268.45 s 
+2024-09-05 04:28:51.055656: Yayy! New best EMA pseudo Dice: 0.9387 
+2024-09-05 04:29:04.078850:  
+2024-09-05 04:29:04.079699: Epoch 96 
+2024-09-05 04:29:04.080369: Current learning rate: 0.00913 
+2024-09-05 04:33:59.188837: train_loss -0.7071 
+2024-09-05 04:33:59.200398: val_loss -0.6945 
+2024-09-05 04:33:59.201288: Pseudo dice [0.9547, 0.9446, 0.9766, 0.9375, 0.8315, 0.9528, 0.9877] 
+2024-09-05 04:33:59.202234: Epoch time: 295.11 s 
+2024-09-05 04:33:59.202987: Yayy! New best EMA pseudo Dice: 0.9389 
+2024-09-05 04:34:09.635742:  
+2024-09-05 04:34:09.636971: Epoch 97 
+2024-09-05 04:34:09.637798: Current learning rate: 0.00912 
+2024-09-05 04:38:53.084657: train_loss -0.7068 
+2024-09-05 04:38:53.116046: val_loss -0.7039 
+2024-09-05 04:38:53.117336: Pseudo dice [0.9538, 0.9516, 0.97, 0.9331, 0.8391, 0.9536, 0.9879] 
+2024-09-05 04:38:53.118927: Epoch time: 283.45 s 
+2024-09-05 04:38:53.120070: Yayy! New best EMA pseudo Dice: 0.9392 
+2024-09-05 04:39:02.452936:  
+2024-09-05 04:39:02.454643: Epoch 98 
+2024-09-05 04:39:02.457238: Current learning rate: 0.00911 
+2024-09-05 04:43:36.117567: train_loss -0.7181 
+2024-09-05 04:43:36.119013: val_loss -0.7041 
+2024-09-05 04:43:36.119990: Pseudo dice [0.9554, 0.9458, 0.9733, 0.9304, 0.839, 0.9542, 0.9873] 
+2024-09-05 04:43:36.120871: Epoch time: 273.67 s 
+2024-09-05 04:43:36.121834: Yayy! New best EMA pseudo Dice: 0.9393 
+2024-09-05 04:43:45.331787:  
+2024-09-05 04:43:45.332751: Epoch 99 
+2024-09-05 04:43:45.333519: Current learning rate: 0.0091 
+2024-09-05 04:48:36.631470: train_loss -0.6965 
+2024-09-05 04:48:36.647426: val_loss -0.7389 
+2024-09-05 04:48:36.648618: Pseudo dice [0.9594, 0.9513, 0.9743, 0.9423, 0.8439, 0.9575, 0.9883] 
+2024-09-05 04:48:36.649562: Epoch time: 291.3 s 
+2024-09-05 04:48:45.144687: Yayy! New best EMA pseudo Dice: 0.9399 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_5_08_31_30.txt

@@ -0,0 +1,394 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-05 08:31:33.791886: Using torch.compile... 
+2024-09-05 08:31:42.736804: do_dummy_2d_data_aug: False 
+2024-09-05 08:31:42.742971: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-05 08:31:42.744489: The split file contains 5 splits. 
+2024-09-05 08:31:42.745110: Desired fold for training: 0 
+2024-09-05 08:31:42.745684: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-05 08:31:54.278436: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-05 08:31:55.938057:  
+2024-09-05 08:31:55.950758: Epoch 100 
+2024-09-05 08:31:55.952600: Current learning rate: 0.0091 
+2024-09-05 08:50:27.089407: train_loss -0.705 
+2024-09-05 08:50:27.090862: val_loss -0.6685 
+2024-09-05 08:50:27.091622: Pseudo dice [0.9526, 0.9471, 0.9695, 0.937, 0.8358, 0.9562, 0.9845] 
+2024-09-05 08:50:27.092400: Epoch time: 1111.16 s 
+2024-09-05 08:50:27.093047: Yayy! New best EMA pseudo Dice: 0.94 
+2024-09-05 08:50:34.912471:  
+2024-09-05 08:50:34.913294: Epoch 101 
+2024-09-05 08:50:34.913781: Current learning rate: 0.00909 
+2024-09-05 08:54:58.876535: train_loss -0.6965 
+2024-09-05 08:54:58.887389: val_loss -0.6913 
+2024-09-05 08:54:58.888862: Pseudo dice [0.9546, 0.9451, 0.9696, 0.9373, 0.8299, 0.9558, 0.9876] 
+2024-09-05 08:54:58.890111: Epoch time: 263.97 s 
+2024-09-05 08:54:58.892781: Yayy! New best EMA pseudo Dice: 0.94 
+2024-09-05 08:55:07.111125:  
+2024-09-05 08:55:07.111877: Epoch 102 
+2024-09-05 08:55:07.112417: Current learning rate: 0.00908 
+2024-09-05 08:59:41.146031: train_loss -0.707 
+2024-09-05 08:59:41.147572: val_loss -0.6448 
+2024-09-05 08:59:41.148183: Pseudo dice [0.9502, 0.9462, 0.968, 0.9327, 0.8261, 0.9068, 0.9875] 
+2024-09-05 08:59:41.148826: Epoch time: 274.04 s 
+2024-09-05 08:59:43.658873:  
+2024-09-05 08:59:43.674195: Epoch 103 
+2024-09-05 08:59:43.675440: Current learning rate: 0.00907 
+2024-09-05 09:04:27.663525: train_loss -0.6848 
+2024-09-05 09:04:27.664976: val_loss -0.7081 
+2024-09-05 09:04:27.665750: Pseudo dice [0.9501, 0.9456, 0.9724, 0.9315, 0.8382, 0.9504, 0.9879] 
+2024-09-05 09:04:27.666523: Epoch time: 284.01 s 
+2024-09-05 09:04:31.611662:  
+2024-09-05 09:04:31.612638: Epoch 104 
+2024-09-05 09:04:31.613331: Current learning rate: 0.00906 
+2024-09-05 09:09:06.190820: train_loss -0.6919 
+2024-09-05 09:09:06.206773: val_loss -0.6759 
+2024-09-05 09:09:06.207633: Pseudo dice [0.9516, 0.9503, 0.9704, 0.932, 0.8528, 0.9518, 0.9869] 
+2024-09-05 09:09:06.208310: Epoch time: 274.58 s 
+2024-09-05 09:09:08.835996:  
+2024-09-05 09:09:08.837058: Epoch 105 
+2024-09-05 09:09:08.837647: Current learning rate: 0.00905 
+2024-09-05 09:13:48.451684: train_loss -0.7018 
+2024-09-05 09:13:48.459122: val_loss -0.6839 
+2024-09-05 09:13:48.460006: Pseudo dice [0.9555, 0.9489, 0.9711, 0.9401, 0.8518, 0.9555, 0.9876] 
+2024-09-05 09:13:48.460622: Epoch time: 279.62 s 
+2024-09-05 09:13:50.949326:  
+2024-09-05 09:13:50.950331: Epoch 106 
+2024-09-05 09:13:50.951338: Current learning rate: 0.00904 
+2024-09-05 09:18:31.255238: train_loss -0.7085 
+2024-09-05 09:18:31.270198: val_loss -0.7061 
+2024-09-05 09:18:31.271889: Pseudo dice [0.9561, 0.9487, 0.9742, 0.9407, 0.8417, 0.9567, 0.9879] 
+2024-09-05 09:18:31.274106: Epoch time: 280.31 s 
+2024-09-05 09:18:31.276635: Yayy! New best EMA pseudo Dice: 0.9403 
+2024-09-05 09:18:39.857884:  
+2024-09-05 09:18:39.858677: Epoch 107 
+2024-09-05 09:18:39.859376: Current learning rate: 0.00903 
+2024-09-05 09:23:12.842974: train_loss -0.6829 
+2024-09-05 09:23:12.870975: val_loss -0.6751 
+2024-09-05 09:23:12.872079: Pseudo dice [0.9551, 0.952, 0.9719, 0.9363, 0.8463, 0.9507, 0.9872] 
+2024-09-05 09:23:12.873170: Epoch time: 272.99 s 
+2024-09-05 09:23:12.873983: Yayy! New best EMA pseudo Dice: 0.9406 
+2024-09-05 09:23:21.194571:  
+2024-09-05 09:23:21.195681: Epoch 108 
+2024-09-05 09:23:21.196999: Current learning rate: 0.00902 
+2024-09-05 09:28:04.830091: train_loss -0.7028 
+2024-09-05 09:28:04.848047: val_loss -0.746 
+2024-09-05 09:28:04.848866: Pseudo dice [0.9557, 0.9463, 0.9727, 0.9319, 0.845, 0.9517, 0.9869] 
+2024-09-05 09:28:04.849694: Epoch time: 283.64 s 
+2024-09-05 09:28:04.850562: Yayy! New best EMA pseudo Dice: 0.9406 
+2024-09-05 09:28:14.028874:  
+2024-09-05 09:28:14.029940: Epoch 109 
+2024-09-05 09:28:14.030728: Current learning rate: 0.00901 
+2024-09-05 09:32:33.565977: train_loss -0.6763 
+2024-09-05 09:32:33.570082: val_loss -0.7217 
+2024-09-05 09:32:33.570839: Pseudo dice [0.9548, 0.9511, 0.9744, 0.9406, 0.847, 0.9567, 0.9878] 
+2024-09-05 09:32:33.571423: Epoch time: 259.54 s 
+2024-09-05 09:32:33.572038: Yayy! New best EMA pseudo Dice: 0.941 
+2024-09-05 09:32:42.279923:  
+2024-09-05 09:32:42.280665: Epoch 110 
+2024-09-05 09:32:42.281219: Current learning rate: 0.009 
+2024-09-05 09:37:12.487998: train_loss -0.7075 
+2024-09-05 09:37:12.551328: val_loss -0.7321 
+2024-09-05 09:37:12.552289: Pseudo dice [0.9566, 0.9488, 0.9745, 0.9397, 0.8472, 0.9602, 0.9892] 
+2024-09-05 09:37:12.553035: Epoch time: 270.21 s 
+2024-09-05 09:37:12.553579: Yayy! New best EMA pseudo Dice: 0.9415 
+2024-09-05 09:37:20.268404:  
+2024-09-05 09:37:20.269186: Epoch 111 
+2024-09-05 09:37:20.269744: Current learning rate: 0.009 
+2024-09-05 09:42:00.148455: train_loss -0.682 
+2024-09-05 09:42:00.167209: val_loss -0.6718 
+2024-09-05 09:42:00.167922: Pseudo dice [0.9515, 0.9479, 0.9678, 0.9332, 0.7979, 0.9513, 0.9853] 
+2024-09-05 09:42:00.168656: Epoch time: 279.88 s 
+2024-09-05 09:42:02.712103:  
+2024-09-05 09:42:02.713485: Epoch 112 
+2024-09-05 09:42:02.714421: Current learning rate: 0.00899 
+2024-09-05 09:46:36.981990: train_loss -0.6893 
+2024-09-05 09:46:36.984857: val_loss -0.7267 
+2024-09-05 09:46:36.985767: Pseudo dice [0.9564, 0.9483, 0.9737, 0.936, 0.8547, 0.9593, 0.9893] 
+2024-09-05 09:46:36.986682: Epoch time: 274.27 s 
+2024-09-05 09:46:39.408468:  
+2024-09-05 09:46:39.409366: Epoch 113 
+2024-09-05 09:46:39.410108: Current learning rate: 0.00898 
+2024-09-05 09:51:34.192128: train_loss -0.6917 
+2024-09-05 09:51:34.216773: val_loss -0.7442 
+2024-09-05 09:51:34.217605: Pseudo dice [0.9562, 0.9428, 0.9709, 0.9422, 0.8439, 0.9527, 0.9866] 
+2024-09-05 09:51:34.218253: Epoch time: 294.79 s 
+2024-09-05 09:51:36.644227:  
+2024-09-05 09:51:36.645267: Epoch 114 
+2024-09-05 09:51:36.645923: Current learning rate: 0.00897 
+2024-09-05 09:55:58.862603: train_loss -0.7018 
+2024-09-05 09:55:58.864016: val_loss -0.7099 
+2024-09-05 09:55:58.865019: Pseudo dice [0.9546, 0.9492, 0.9731, 0.9353, 0.8349, 0.9581, 0.9875] 
+2024-09-05 09:55:58.865691: Epoch time: 262.22 s 
+2024-09-05 09:56:02.911829:  
+2024-09-05 09:56:02.926658: Epoch 115 
+2024-09-05 09:56:02.927508: Current learning rate: 0.00896 
+2024-09-05 10:00:43.608797: train_loss -0.6916 
+2024-09-05 10:00:43.616673: val_loss -0.7027 
+2024-09-05 10:00:43.618003: Pseudo dice [0.9583, 0.9525, 0.9754, 0.9403, 0.843, 0.9602, 0.9892] 
+2024-09-05 10:00:43.619066: Epoch time: 280.7 s 
+2024-09-05 10:00:43.619881: Yayy! New best EMA pseudo Dice: 0.9417 
+2024-09-05 10:00:52.367775:  
+2024-09-05 10:00:52.368668: Epoch 116 
+2024-09-05 10:00:52.369438: Current learning rate: 0.00895 
+2024-09-05 10:05:33.767412: train_loss -0.7022 
+2024-09-05 10:05:33.769071: val_loss -0.744 
+2024-09-05 10:05:33.786280: Pseudo dice [0.9566, 0.9484, 0.9746, 0.9405, 0.8532, 0.9384, 0.9892] 
+2024-09-05 10:05:33.794727: Epoch time: 281.4 s 
+2024-09-05 10:05:33.795464: Yayy! New best EMA pseudo Dice: 0.9419 
+2024-09-05 10:05:42.262891:  
+2024-09-05 10:05:42.263763: Epoch 117 
+2024-09-05 10:05:42.264347: Current learning rate: 0.00894 
+2024-09-05 10:10:09.350181: train_loss -0.7033 
+2024-09-05 10:10:09.352794: val_loss -0.7001 
+2024-09-05 10:10:09.353674: Pseudo dice [0.9576, 0.9509, 0.9752, 0.9408, 0.8573, 0.9526, 0.9889] 
+2024-09-05 10:10:09.369806: Epoch time: 267.09 s 
+2024-09-05 10:10:09.386639: Yayy! New best EMA pseudo Dice: 0.9423 
+2024-09-05 10:10:19.512285:  
+2024-09-05 10:10:19.513151: Epoch 118 
+2024-09-05 10:10:19.513940: Current learning rate: 0.00893 
+2024-09-05 10:14:57.567624: train_loss -0.7132 
+2024-09-05 10:14:57.569076: val_loss -0.7213 
+2024-09-05 10:14:57.569889: Pseudo dice [0.9609, 0.9533, 0.9768, 0.941, 0.8507, 0.9622, 0.9857] 
+2024-09-05 10:14:57.570544: Epoch time: 278.06 s 
+2024-09-05 10:14:57.571515: Yayy! New best EMA pseudo Dice: 0.9428 
+2024-09-05 10:15:06.580484:  
+2024-09-05 10:15:06.581460: Epoch 119 
+2024-09-05 10:15:06.582438: Current learning rate: 0.00892 
+2024-09-05 10:19:50.665343: train_loss -0.718 
+2024-09-05 10:19:50.668527: val_loss -0.7147 
+2024-09-05 10:19:50.669376: Pseudo dice [0.9535, 0.9497, 0.9709, 0.925, 0.8323, 0.9577, 0.9896] 
+2024-09-05 10:19:50.672900: Epoch time: 284.09 s 
+2024-09-05 10:19:53.484023:  
+2024-09-05 10:19:53.484874: Epoch 120 
+2024-09-05 10:19:53.485564: Current learning rate: 0.00891 
+2024-09-05 10:24:37.565263: train_loss -0.7075 
+2024-09-05 10:24:37.579084: val_loss -0.7525 
+2024-09-05 10:24:37.580091: Pseudo dice [0.9579, 0.9521, 0.9737, 0.9399, 0.8477, 0.958, 0.9876] 
+2024-09-05 10:24:37.580788: Epoch time: 284.09 s 
+2024-09-05 10:24:39.793303:  
+2024-09-05 10:24:39.795532: Epoch 121 
+2024-09-05 10:24:39.796145: Current learning rate: 0.0089 
+2024-09-05 10:29:18.214857: train_loss -0.6912 
+2024-09-05 10:29:18.239968: val_loss -0.7215 
+2024-09-05 10:29:18.241069: Pseudo dice [0.9553, 0.949, 0.9725, 0.9308, 0.8399, 0.9581, 0.9884] 
+2024-09-05 10:29:18.241995: Epoch time: 278.43 s 
+2024-09-05 10:29:20.725494:  
+2024-09-05 10:29:20.726389: Epoch 122 
+2024-09-05 10:29:20.727086: Current learning rate: 0.00889 
+2024-09-05 10:33:55.156586: train_loss -0.7059 
+2024-09-05 10:33:55.166357: val_loss -0.7166 
+2024-09-05 10:33:55.167267: Pseudo dice [0.9537, 0.9465, 0.9705, 0.9387, 0.8309, 0.9501, 0.9857] 
+2024-09-05 10:33:55.168248: Epoch time: 274.44 s 
+2024-09-05 10:33:57.671613:  
+2024-09-05 10:33:57.689941: Epoch 123 
+2024-09-05 10:33:57.690939: Current learning rate: 0.00889 
+2024-09-05 10:38:21.012477: train_loss -0.7044 
+2024-09-05 10:38:21.025206: val_loss -0.703 
+2024-09-05 10:38:21.025988: Pseudo dice [0.9576, 0.95, 0.9739, 0.9344, 0.8346, 0.9559, 0.9867] 
+2024-09-05 10:38:21.026950: Epoch time: 263.36 s 
+2024-09-05 10:38:24.426492:  
+2024-09-05 10:38:24.427408: Epoch 124 
+2024-09-05 10:38:24.428055: Current learning rate: 0.00888 
+2024-09-05 10:43:18.964696: train_loss -0.7021 
+2024-09-05 10:43:18.969211: val_loss -0.7356 
+2024-09-05 10:43:18.970610: Pseudo dice [0.9578, 0.9528, 0.9746, 0.937, 0.8332, 0.9494, 0.9876] 
+2024-09-05 10:43:18.971966: Epoch time: 294.54 s 
+2024-09-05 10:43:21.257270:  
+2024-09-05 10:43:21.258263: Epoch 125 
+2024-09-05 10:43:21.259050: Current learning rate: 0.00887 
+2024-09-05 10:47:45.710503: train_loss -0.7211 
+2024-09-05 10:47:45.711966: val_loss -0.7172 
+2024-09-05 10:47:45.712581: Pseudo dice [0.9585, 0.9514, 0.9746, 0.934, 0.8453, 0.955, 0.988] 
+2024-09-05 10:47:45.713318: Epoch time: 264.46 s 
+2024-09-05 10:47:48.264259:  
+2024-09-05 10:47:48.265147: Epoch 126 
+2024-09-05 10:47:48.265873: Current learning rate: 0.00886 
+2024-09-05 10:52:34.509533: train_loss -0.7119 
+2024-09-05 10:52:34.511611: val_loss -0.686 
+2024-09-05 10:52:34.513368: Pseudo dice [0.9601, 0.9511, 0.9762, 0.9387, 0.83, 0.954, 0.9882] 
+2024-09-05 10:52:34.514530: Epoch time: 286.25 s 
+2024-09-05 10:52:38.065735:  
+2024-09-05 10:52:38.066900: Epoch 127 
+2024-09-05 10:52:38.067740: Current learning rate: 0.00885 
+2024-09-05 10:57:21.281061: train_loss -0.7086 
+2024-09-05 10:57:21.292991: val_loss -0.6917 
+2024-09-05 10:57:21.294309: Pseudo dice [0.9524, 0.9476, 0.9689, 0.9371, 0.8341, 0.9529, 0.9855] 
+2024-09-05 10:57:21.295094: Epoch time: 283.22 s 
+2024-09-05 10:57:24.093887:  
+2024-09-05 10:57:24.094779: Epoch 128 
+2024-09-05 10:57:24.095411: Current learning rate: 0.00884 
+2024-09-05 11:02:19.833550: train_loss -0.6924 
+2024-09-05 11:02:19.851095: val_loss -0.7199 
+2024-09-05 11:02:19.858641: Pseudo dice [0.952, 0.9486, 0.9675, 0.9366, 0.8449, 0.9583, 0.9845] 
+2024-09-05 11:02:19.862434: Epoch time: 295.74 s 
+2024-09-05 11:02:22.249124:  
+2024-09-05 11:02:22.251416: Epoch 129 
+2024-09-05 11:02:22.253581: Current learning rate: 0.00883 
+2024-09-05 11:07:10.335835: train_loss -0.7206 
+2024-09-05 11:07:10.337374: val_loss -0.7266 
+2024-09-05 11:07:10.338109: Pseudo dice [0.9594, 0.9548, 0.9772, 0.943, 0.8568, 0.9617, 0.9858] 
+2024-09-05 11:07:10.339780: Epoch time: 288.09 s 
+2024-09-05 11:07:12.672329:  
+2024-09-05 11:07:12.673326: Epoch 130 
+2024-09-05 11:07:12.673888: Current learning rate: 0.00882 
+2024-09-05 11:11:55.963713: train_loss -0.7191 
+2024-09-05 11:11:55.990924: val_loss -0.7081 
+2024-09-05 11:11:55.991768: Pseudo dice [0.9526, 0.9481, 0.9676, 0.9464, 0.8561, 0.9543, 0.9876] 
+2024-09-05 11:11:55.992546: Epoch time: 283.3 s 
+2024-09-05 11:11:55.993223: Yayy! New best EMA pseudo Dice: 0.943 
+2024-09-05 11:12:04.119151:  
+2024-09-05 11:12:04.121111: Epoch 131 
+2024-09-05 11:12:04.123664: Current learning rate: 0.00881 
+2024-09-05 11:16:24.946059: train_loss -0.7158 
+2024-09-05 11:16:24.963580: val_loss -0.7443 
+2024-09-05 11:16:24.964913: Pseudo dice [0.9584, 0.9509, 0.9739, 0.9336, 0.8453, 0.9511, 0.9877] 
+2024-09-05 11:16:24.966341: Epoch time: 260.83 s 
+2024-09-05 11:16:24.967621: Yayy! New best EMA pseudo Dice: 0.943 
+2024-09-05 11:16:33.134722:  
+2024-09-05 11:16:33.135606: Epoch 132 
+2024-09-05 11:16:33.136261: Current learning rate: 0.0088 
+2024-09-05 11:21:00.147625: train_loss -0.6902 
+2024-09-05 11:21:00.158601: val_loss -0.7335 
+2024-09-05 11:21:00.159343: Pseudo dice [0.9599, 0.9512, 0.977, 0.9377, 0.8551, 0.9629, 0.9887] 
+2024-09-05 11:21:00.160099: Epoch time: 267.02 s 
+2024-09-05 11:21:00.160653: Yayy! New best EMA pseudo Dice: 0.9434 
+2024-09-05 11:21:09.873146:  
+2024-09-05 11:21:09.873924: Epoch 133 
+2024-09-05 11:21:09.874502: Current learning rate: 0.00879 
+2024-09-05 11:25:28.391029: train_loss -0.7111 
+2024-09-05 11:25:28.424236: val_loss -0.6916 
+2024-09-05 11:25:28.425438: Pseudo dice [0.9563, 0.9519, 0.9716, 0.9393, 0.844, 0.9568, 0.9889] 
+2024-09-05 11:25:28.426058: Epoch time: 258.52 s 
+2024-09-05 11:25:28.426905: Yayy! New best EMA pseudo Dice: 0.9435 
+2024-09-05 11:25:37.920934:  
+2024-09-05 11:25:37.921854: Epoch 134 
+2024-09-05 11:25:37.922633: Current learning rate: 0.00879 
+2024-09-05 11:30:09.721164: train_loss -0.7161 
+2024-09-05 11:30:09.723393: val_loss -0.7005 
+2024-09-05 11:30:09.724198: Pseudo dice [0.957, 0.9514, 0.9742, 0.9313, 0.8338, 0.9546, 0.9881] 
+2024-09-05 11:30:09.724806: Epoch time: 271.81 s 
+2024-09-05 11:30:11.882450:  
+2024-09-05 11:30:11.883672: Epoch 135 
+2024-09-05 11:30:11.884293: Current learning rate: 0.00878 
+2024-09-05 11:34:43.952901: train_loss -0.7221 
+2024-09-05 11:34:43.956115: val_loss -0.7002 
+2024-09-05 11:34:43.957419: Pseudo dice [0.9598, 0.9518, 0.9733, 0.9376, 0.8461, 0.9612, 0.9889] 
+2024-09-05 11:34:43.958086: Epoch time: 272.08 s 
+2024-09-05 11:34:43.959118: Yayy! New best EMA pseudo Dice: 0.9435 
+2024-09-05 11:34:52.033961:  
+2024-09-05 11:34:52.034847: Epoch 136 
+2024-09-05 11:34:52.035497: Current learning rate: 0.00877 
+2024-09-05 11:39:39.139255: train_loss -0.7103 
+2024-09-05 11:39:39.141555: val_loss -0.7127 
+2024-09-05 11:39:39.142130: Pseudo dice [0.9582, 0.9502, 0.9762, 0.9387, 0.8391, 0.959, 0.9878] 
+2024-09-05 11:39:39.142875: Epoch time: 287.11 s 
+2024-09-05 11:39:39.143398: Yayy! New best EMA pseudo Dice: 0.9436 
+2024-09-05 11:39:47.289981:  
+2024-09-05 11:39:47.290783: Epoch 137 
+2024-09-05 11:39:47.291403: Current learning rate: 0.00876 
+2024-09-05 11:44:18.743298: train_loss -0.7179 
+2024-09-05 11:44:18.746574: val_loss -0.7201 
+2024-09-05 11:44:18.747404: Pseudo dice [0.9565, 0.9517, 0.973, 0.9384, 0.8491, 0.9573, 0.9897] 
+2024-09-05 11:44:18.748511: Epoch time: 271.46 s 
+2024-09-05 11:44:18.749377: Yayy! New best EMA pseudo Dice: 0.9437 
+2024-09-05 11:44:28.894542:  
+2024-09-05 11:44:28.895589: Epoch 138 
+2024-09-05 11:44:28.896357: Current learning rate: 0.00875 
+2024-09-05 11:49:05.269133: train_loss -0.7117 
+2024-09-05 11:49:05.272372: val_loss -0.7382 
+2024-09-05 11:49:05.273381: Pseudo dice [0.9584, 0.9506, 0.9748, 0.94, 0.8536, 0.9585, 0.9887] 
+2024-09-05 11:49:05.274408: Epoch time: 276.38 s 
+2024-09-05 11:49:05.275112: Yayy! New best EMA pseudo Dice: 0.944 
+2024-09-05 11:49:12.800072:  
+2024-09-05 11:49:12.801045: Epoch 139 
+2024-09-05 11:49:12.801717: Current learning rate: 0.00874 
+2024-09-05 11:53:37.600915: train_loss -0.7212 
+2024-09-05 11:53:37.603848: val_loss -0.7139 
+2024-09-05 11:53:37.604887: Pseudo dice [0.9586, 0.953, 0.9737, 0.9468, 0.8557, 0.9573, 0.9901] 
+2024-09-05 11:53:37.606223: Epoch time: 264.8 s 
+2024-09-05 11:53:37.607081: Yayy! New best EMA pseudo Dice: 0.9444 
+2024-09-05 11:53:45.968114:  
+2024-09-05 11:53:45.969033: Epoch 140 
+2024-09-05 11:53:45.969635: Current learning rate: 0.00873 
+2024-09-05 11:58:19.584594: train_loss -0.7263 
+2024-09-05 11:58:19.587180: val_loss -0.7109 
+2024-09-05 11:58:19.587955: Pseudo dice [0.9579, 0.9503, 0.9692, 0.9404, 0.867, 0.956, 0.9901] 
+2024-09-05 11:58:19.588639: Epoch time: 273.62 s 
+2024-09-05 11:58:19.589290: Yayy! New best EMA pseudo Dice: 0.9447 
+2024-09-05 11:58:27.613800:  
+2024-09-05 11:58:27.614672: Epoch 141 
+2024-09-05 11:58:27.615329: Current learning rate: 0.00872 
+2024-09-05 12:03:11.264485: train_loss -0.7151 
+2024-09-05 12:03:11.266899: val_loss -0.7137 
+2024-09-05 12:03:11.267687: Pseudo dice [0.9497, 0.9454, 0.9711, 0.9354, 0.8487, 0.9547, 0.9902] 
+2024-09-05 12:03:11.268525: Epoch time: 283.65 s 
+2024-09-05 12:03:15.750800:  
+2024-09-05 12:03:15.752442: Epoch 142 
+2024-09-05 12:03:15.753248: Current learning rate: 0.00871 
+2024-09-05 12:07:49.836756: train_loss -0.6953 
+2024-09-05 12:07:49.839632: val_loss -0.7252 
+2024-09-05 12:07:49.840572: Pseudo dice [0.9553, 0.9488, 0.9746, 0.9261, 0.8307, 0.9518, 0.989] 
+2024-09-05 12:07:49.841883: Epoch time: 274.09 s 
+2024-09-05 12:07:53.073098:  
+2024-09-05 12:07:53.073984: Epoch 143 
+2024-09-05 12:07:53.074636: Current learning rate: 0.0087 
+2024-09-05 12:12:35.461232: train_loss -0.7052 
+2024-09-05 12:12:35.469450: val_loss -0.7183 
+2024-09-05 12:12:35.470131: Pseudo dice [0.9568, 0.9484, 0.9739, 0.9466, 0.8531, 0.9577, 0.9897] 
+2024-09-05 12:12:35.470809: Epoch time: 282.39 s 
+2024-09-05 12:12:38.272817:  
+2024-09-05 12:12:38.273803: Epoch 144 
+2024-09-05 12:12:38.274493: Current learning rate: 0.00869 
+2024-09-05 12:17:15.281117: train_loss -0.7138 
+2024-09-05 12:17:15.282886: val_loss -0.7262 
+2024-09-05 12:17:15.283513: Pseudo dice [0.9584, 0.9538, 0.9741, 0.9492, 0.8555, 0.9603, 0.9904] 
+2024-09-05 12:17:15.284355: Epoch time: 277.01 s 
+2024-09-05 12:17:17.615475:  
+2024-09-05 12:17:17.616538: Epoch 145 
+2024-09-05 12:17:17.617204: Current learning rate: 0.00868 
+2024-09-05 12:21:58.237548: train_loss -0.7172 
+2024-09-05 12:21:58.256968: val_loss -0.7144 
+2024-09-05 12:21:58.257753: Pseudo dice [0.9572, 0.9536, 0.9742, 0.9384, 0.8556, 0.9595, 0.9899] 
+2024-09-05 12:21:58.258553: Epoch time: 280.63 s 
+2024-09-05 12:21:58.259136: Yayy! New best EMA pseudo Dice: 0.9449 
+2024-09-05 12:22:06.673335:  
+2024-09-05 12:22:06.674195: Epoch 146 
+2024-09-05 12:22:06.674786: Current learning rate: 0.00868 
+2024-09-05 12:26:59.060425: train_loss -0.6983 
+2024-09-05 12:26:59.062918: val_loss -0.723 
+2024-09-05 12:26:59.064022: Pseudo dice [0.9578, 0.9483, 0.9764, 0.9417, 0.8327, 0.9576, 0.987] 
+2024-09-05 12:26:59.065417: Epoch time: 292.39 s 
+2024-09-05 12:27:02.050327:  
+2024-09-05 12:27:02.061985: Epoch 147 
+2024-09-05 12:27:02.063771: Current learning rate: 0.00867 
+2024-09-05 12:31:29.781280: train_loss -0.7134 
+2024-09-05 12:31:29.795758: val_loss -0.72 
+2024-09-05 12:31:29.796530: Pseudo dice [0.9566, 0.9496, 0.9748, 0.9414, 0.8483, 0.9607, 0.9886] 
+2024-09-05 12:31:29.797140: Epoch time: 267.79 s 
+2024-09-05 12:31:32.550988:  
+2024-09-05 12:31:32.552139: Epoch 148 
+2024-09-05 12:31:32.553009: Current learning rate: 0.00866 
+2024-09-05 12:36:25.971860: train_loss -0.7088 
+2024-09-05 12:36:25.981588: val_loss -0.7358 
+2024-09-05 12:36:25.982729: Pseudo dice [0.9581, 0.9479, 0.9719, 0.9401, 0.865, 0.9607, 0.9898] 
+2024-09-05 12:36:25.984284: Epoch time: 293.43 s 
+2024-09-05 12:36:25.985542: Yayy! New best EMA pseudo Dice: 0.9451 
+2024-09-05 12:36:34.056867:  
+2024-09-05 12:36:34.057711: Epoch 149 
+2024-09-05 12:36:34.058443: Current learning rate: 0.00865 
+2024-09-05 12:41:04.382096: train_loss -0.7064 
+2024-09-05 12:41:04.384546: val_loss -0.6911 
+2024-09-05 12:41:04.386032: Pseudo dice [0.9553, 0.9509, 0.974, 0.9424, 0.841, 0.9614, 0.9861] 
+2024-09-05 12:41:04.386887: Epoch time: 270.33 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_5_16_30_45.txt

@@ -0,0 +1,381 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-05 16:30:48.114985: Using torch.compile... 
+2024-09-05 16:30:57.262532: do_dummy_2d_data_aug: False 
+2024-09-05 16:30:57.268567: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-05 16:30:57.269850: The split file contains 5 splits. 
+2024-09-05 16:30:57.270423: Desired fold for training: 0 
+2024-09-05 16:30:57.271030: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-05 16:31:08.556564: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-05 16:31:10.073337:  
+2024-09-05 16:31:10.075511: Epoch 150 
+2024-09-05 16:31:10.078360: Current learning rate: 0.00864 
+2024-09-05 16:49:44.442162: train_loss -0.7122 
+2024-09-05 16:49:44.443352: val_loss -0.7159 
+2024-09-05 16:49:44.444150: Pseudo dice [0.9567, 0.9515, 0.9696, 0.9372, 0.845, 0.9574, 0.9871] 
+2024-09-05 16:49:44.444802: Epoch time: 1114.37 s 
+2024-09-05 16:49:46.112066:  
+2024-09-05 16:49:46.223099: Epoch 151 
+2024-09-05 16:49:46.331435: Current learning rate: 0.00863 
+2024-09-05 16:54:29.417373: train_loss -0.7305 
+2024-09-05 16:54:29.429139: val_loss -0.7137 
+2024-09-05 16:54:29.429954: Pseudo dice [0.9586, 0.9538, 0.9743, 0.9428, 0.8347, 0.9527, 0.9863] 
+2024-09-05 16:54:29.430523: Epoch time: 283.31 s 
+2024-09-05 16:54:31.797340:  
+2024-09-05 16:54:31.798274: Epoch 152 
+2024-09-05 16:54:31.799184: Current learning rate: 0.00862 
+2024-09-05 16:59:20.116929: train_loss -0.7374 
+2024-09-05 16:59:20.118404: val_loss -0.7409 
+2024-09-05 16:59:20.119238: Pseudo dice [0.9616, 0.9512, 0.977, 0.9382, 0.8517, 0.9619, 0.9908] 
+2024-09-05 16:59:20.120194: Epoch time: 288.32 s 
+2024-09-05 16:59:23.341870:  
+2024-09-05 16:59:23.342713: Epoch 153 
+2024-09-05 16:59:23.343449: Current learning rate: 0.00861 
+2024-09-05 17:03:53.803570: train_loss -0.713 
+2024-09-05 17:03:53.814083: val_loss -0.7026 
+2024-09-05 17:03:53.814965: Pseudo dice [0.9585, 0.9509, 0.9753, 0.9376, 0.8424, 0.9513, 0.9868] 
+2024-09-05 17:03:53.815760: Epoch time: 270.47 s 
+2024-09-05 17:03:56.688178:  
+2024-09-05 17:03:56.696356: Epoch 154 
+2024-09-05 17:03:56.697041: Current learning rate: 0.0086 
+2024-09-05 17:08:27.807127: train_loss -0.7249 
+2024-09-05 17:08:27.809022: val_loss -0.7449 
+2024-09-05 17:08:27.810742: Pseudo dice [0.9592, 0.9495, 0.9762, 0.9331, 0.8658, 0.962, 0.9913] 
+2024-09-05 17:08:27.812331: Epoch time: 271.12 s 
+2024-09-05 17:08:27.813962: Yayy! New best EMA pseudo Dice: 0.9451 
+2024-09-05 17:08:42.729360:  
+2024-09-05 17:08:42.781580: Epoch 155 
+2024-09-05 17:08:42.843530: Current learning rate: 0.00859 
+2024-09-05 17:13:36.178315: train_loss -0.715 
+2024-09-05 17:13:36.180984: val_loss -0.7043 
+2024-09-05 17:13:36.181750: Pseudo dice [0.959, 0.9545, 0.9727, 0.9451, 0.8557, 0.9621, 0.9875] 
+2024-09-05 17:13:36.182761: Epoch time: 293.45 s 
+2024-09-05 17:13:36.183593: Yayy! New best EMA pseudo Dice: 0.9454 
+2024-09-05 17:13:45.053745:  
+2024-09-05 17:13:45.054882: Epoch 156 
+2024-09-05 17:13:45.055664: Current learning rate: 0.00858 
+2024-09-05 17:18:32.228349: train_loss -0.7139 
+2024-09-05 17:18:32.240942: val_loss -0.6976 
+2024-09-05 17:18:32.241735: Pseudo dice [0.9554, 0.9479, 0.9751, 0.9431, 0.8535, 0.9629, 0.9897] 
+2024-09-05 17:18:32.242606: Epoch time: 287.18 s 
+2024-09-05 17:18:32.243301: Yayy! New best EMA pseudo Dice: 0.9456 
+2024-09-05 17:18:43.507134:  
+2024-09-05 17:18:43.508090: Epoch 157 
+2024-09-05 17:18:43.509127: Current learning rate: 0.00858 
+2024-09-05 17:23:44.457680: train_loss -0.7084 
+2024-09-05 17:23:44.459941: val_loss -0.6921 
+2024-09-05 17:23:44.460915: Pseudo dice [0.9519, 0.9496, 0.9691, 0.9391, 0.8332, 0.9602, 0.9877] 
+2024-09-05 17:23:44.461725: Epoch time: 300.96 s 
+2024-09-05 17:23:47.816007:  
+2024-09-05 17:23:47.821062: Epoch 158 
+2024-09-05 17:23:47.830581: Current learning rate: 0.00857 
+2024-09-05 17:29:01.012879: train_loss -0.709 
+2024-09-05 17:29:01.028203: val_loss -0.7347 
+2024-09-05 17:29:01.029303: Pseudo dice [0.9581, 0.9499, 0.9661, 0.9379, 0.8532, 0.9601, 0.9898] 
+2024-09-05 17:29:01.030416: Epoch time: 313.2 s 
+2024-09-05 17:29:03.948863:  
+2024-09-05 17:29:03.950119: Epoch 159 
+2024-09-05 17:29:03.950909: Current learning rate: 0.00856 
+2024-09-05 17:34:14.904611: train_loss -0.7164 
+2024-09-05 17:34:14.907138: val_loss -0.7323 
+2024-09-05 17:34:14.908109: Pseudo dice [0.9592, 0.9497, 0.9764, 0.9404, 0.8513, 0.9643, 0.9877] 
+2024-09-05 17:34:14.908777: Epoch time: 310.96 s 
+2024-09-05 17:34:17.721457:  
+2024-09-05 17:34:17.722587: Epoch 160 
+2024-09-05 17:34:17.723581: Current learning rate: 0.00855 
+2024-09-05 17:39:27.472079: train_loss -0.7276 
+2024-09-05 17:39:27.478722: val_loss -0.7148 
+2024-09-05 17:39:27.480130: Pseudo dice [0.9555, 0.9517, 0.9719, 0.9384, 0.8549, 0.9587, 0.9902] 
+2024-09-05 17:39:27.481592: Epoch time: 309.76 s 
+2024-09-05 17:39:31.826578:  
+2024-09-05 17:39:31.828084: Epoch 161 
+2024-09-05 17:39:31.829499: Current learning rate: 0.00854 
+2024-09-05 17:44:47.105471: train_loss -0.7189 
+2024-09-05 17:44:47.107126: val_loss -0.7345 
+2024-09-05 17:44:47.107942: Pseudo dice [0.9569, 0.9489, 0.9736, 0.9398, 0.8561, 0.9602, 0.9878] 
+2024-09-05 17:44:47.108828: Epoch time: 315.28 s 
+2024-09-05 17:44:49.884308:  
+2024-09-05 17:44:49.885559: Epoch 162 
+2024-09-05 17:44:49.886342: Current learning rate: 0.00853 
+2024-09-05 17:50:11.903840: train_loss -0.7064 
+2024-09-05 17:50:11.916186: val_loss -0.7159 
+2024-09-05 17:50:11.917502: Pseudo dice [0.9598, 0.9551, 0.9742, 0.9329, 0.8521, 0.9587, 0.9883] 
+2024-09-05 17:50:11.921231: Epoch time: 322.03 s 
+2024-09-05 17:50:14.889348:  
+2024-09-05 17:50:14.891330: Epoch 163 
+2024-09-05 17:50:14.893092: Current learning rate: 0.00852 
+2024-09-05 17:55:38.135309: train_loss -0.7161 
+2024-09-05 17:55:38.137069: val_loss -0.714 
+2024-09-05 17:55:38.138048: Pseudo dice [0.9516, 0.9516, 0.9729, 0.9345, 0.8592, 0.9545, 0.9891] 
+2024-09-05 17:55:38.138751: Epoch time: 323.25 s 
+2024-09-05 17:55:40.861348:  
+2024-09-05 17:55:40.862534: Epoch 164 
+2024-09-05 17:55:40.863469: Current learning rate: 0.00851 
+2024-09-05 18:00:59.919786: train_loss -0.7106 
+2024-09-05 18:00:59.936834: val_loss -0.7279 
+2024-09-05 18:00:59.938690: Pseudo dice [0.9587, 0.9536, 0.9742, 0.941, 0.849, 0.9599, 0.9893] 
+2024-09-05 18:00:59.940453: Epoch time: 319.06 s 
+2024-09-05 18:01:02.883392:  
+2024-09-05 18:01:02.885017: Epoch 165 
+2024-09-05 18:01:02.895176: Current learning rate: 0.0085 
+2024-09-05 18:06:03.292455: train_loss -0.7075 
+2024-09-05 18:06:03.302773: val_loss -0.7156 
+2024-09-05 18:06:03.303597: Pseudo dice [0.9583, 0.9526, 0.9728, 0.9438, 0.8318, 0.9563, 0.9882] 
+2024-09-05 18:06:03.304214: Epoch time: 300.41 s 
+2024-09-05 18:06:06.247953:  
+2024-09-05 18:06:06.248957: Epoch 166 
+2024-09-05 18:06:06.249748: Current learning rate: 0.00849 
+2024-09-05 18:11:29.421145: train_loss -0.7283 
+2024-09-05 18:11:29.431584: val_loss -0.7297 
+2024-09-05 18:11:29.432477: Pseudo dice [0.9597, 0.9537, 0.9774, 0.9425, 0.8623, 0.9658, 0.9904] 
+2024-09-05 18:11:29.433407: Epoch time: 323.18 s 
+2024-09-05 18:11:29.434325: Yayy! New best EMA pseudo Dice: 0.9458 
+2024-09-05 18:11:40.926808:  
+2024-09-05 18:11:40.927939: Epoch 167 
+2024-09-05 18:11:40.928674: Current learning rate: 0.00848 
+2024-09-05 18:16:32.755958: train_loss -0.7146 
+2024-09-05 18:16:32.763396: val_loss -0.7215 
+2024-09-05 18:16:32.764482: Pseudo dice [0.9588, 0.9505, 0.9745, 0.9386, 0.845, 0.9615, 0.9894] 
+2024-09-05 18:16:32.765835: Epoch time: 291.83 s 
+2024-09-05 18:16:35.800260:  
+2024-09-05 18:16:35.802013: Epoch 168 
+2024-09-05 18:16:35.803008: Current learning rate: 0.00847 
+2024-09-05 18:22:18.212695: train_loss -0.7033 
+2024-09-05 18:22:18.223934: val_loss -0.7092 
+2024-09-05 18:22:18.328689: Pseudo dice [0.953, 0.9448, 0.9697, 0.916, 0.8421, 0.9488, 0.9896] 
+2024-09-05 18:22:18.514692: Epoch time: 342.42 s 
+2024-09-05 18:22:21.986162:  
+2024-09-05 18:22:22.158234: Epoch 169 
+2024-09-05 18:22:22.172883: Current learning rate: 0.00847 
+2024-09-05 18:27:37.210751: train_loss -0.6948 
+2024-09-05 18:27:37.213777: val_loss -0.6227 
+2024-09-05 18:27:37.214720: Pseudo dice [0.9392, 0.9363, 0.9576, 0.927, 0.8126, 0.9334, 0.9531] 
+2024-09-05 18:27:37.215740: Epoch time: 315.23 s 
+2024-09-05 18:27:41.455231:  
+2024-09-05 18:27:41.456412: Epoch 170 
+2024-09-05 18:27:41.457230: Current learning rate: 0.00846 
+2024-09-05 18:32:21.131667: train_loss -0.658 
+2024-09-05 18:32:21.136093: val_loss -0.687 
+2024-09-05 18:32:21.138000: Pseudo dice [0.9492, 0.9417, 0.967, 0.9386, 0.8178, 0.9453, 0.9863] 
+2024-09-05 18:32:21.141623: Epoch time: 279.68 s 
+2024-09-05 18:32:24.184405:  
+2024-09-05 18:32:24.201986: Epoch 171 
+2024-09-05 18:32:24.203416: Current learning rate: 0.00845 
+2024-09-05 18:37:18.088207: train_loss -0.6665 
+2024-09-05 18:37:18.090526: val_loss -0.6717 
+2024-09-05 18:37:18.091947: Pseudo dice [0.9487, 0.9459, 0.968, 0.9259, 0.7972, 0.9382, 0.9872] 
+2024-09-05 18:37:18.093262: Epoch time: 293.91 s 
+2024-09-05 18:37:20.779493:  
+2024-09-05 18:37:20.780373: Epoch 172 
+2024-09-05 18:37:20.781380: Current learning rate: 0.00844 
+2024-09-05 18:42:35.502051: train_loss -0.6705 
+2024-09-05 18:42:35.519377: val_loss -0.7124 
+2024-09-05 18:42:35.520105: Pseudo dice [0.957, 0.9476, 0.9749, 0.9235, 0.8256, 0.889, 0.987] 
+2024-09-05 18:42:35.520958: Epoch time: 314.73 s 
+2024-09-05 18:42:38.459105:  
+2024-09-05 18:42:38.460451: Epoch 173 
+2024-09-05 18:42:38.466115: Current learning rate: 0.00843 
+2024-09-05 18:47:54.194916: train_loss -0.7001 
+2024-09-05 18:47:54.204261: val_loss -0.72 
+2024-09-05 18:47:54.207060: Pseudo dice [0.951, 0.9451, 0.9698, 0.9266, 0.8198, 0.9478, 0.9786] 
+2024-09-05 18:47:54.207880: Epoch time: 315.74 s 
+2024-09-05 18:47:56.942095:  
+2024-09-05 18:47:56.942849: Epoch 174 
+2024-09-05 18:47:56.943376: Current learning rate: 0.00842 
+2024-09-05 18:53:07.834568: train_loss -0.7054 
+2024-09-05 18:53:07.837194: val_loss -0.7195 
+2024-09-05 18:53:07.837943: Pseudo dice [0.9518, 0.9466, 0.9666, 0.9225, 0.8416, 0.9548, 0.9882] 
+2024-09-05 18:53:07.838770: Epoch time: 310.9 s 
+2024-09-05 18:53:10.705376:  
+2024-09-05 18:53:10.718719: Epoch 175 
+2024-09-05 18:53:10.719728: Current learning rate: 0.00841 
+2024-09-05 18:58:09.840608: train_loss -0.7052 
+2024-09-05 18:58:09.843425: val_loss -0.6919 
+2024-09-05 18:58:09.844614: Pseudo dice [0.9593, 0.9537, 0.9738, 0.9411, 0.8434, 0.9571, 0.9902] 
+2024-09-05 18:58:09.846097: Epoch time: 299.14 s 
+2024-09-05 18:58:12.611383:  
+2024-09-05 18:58:12.612728: Epoch 176 
+2024-09-05 18:58:12.614429: Current learning rate: 0.0084 
+2024-09-05 19:03:05.981251: train_loss -0.7013 
+2024-09-05 19:03:05.983712: val_loss -0.7603 
+2024-09-05 19:03:05.984639: Pseudo dice [0.9577, 0.9482, 0.977, 0.9399, 0.8422, 0.9588, 0.989] 
+2024-09-05 19:03:05.985269: Epoch time: 293.38 s 
+2024-09-05 19:03:08.788787:  
+2024-09-05 19:03:08.790200: Epoch 177 
+2024-09-05 19:03:08.791935: Current learning rate: 0.00839 
+2024-09-05 19:08:04.795175: train_loss -0.7198 
+2024-09-05 19:08:04.798770: val_loss -0.6857 
+2024-09-05 19:08:04.799721: Pseudo dice [0.9595, 0.954, 0.9746, 0.9363, 0.8578, 0.9612, 0.9866] 
+2024-09-05 19:08:04.801000: Epoch time: 296.01 s 
+2024-09-05 19:08:07.896269:  
+2024-09-05 19:08:07.898165: Epoch 178 
+2024-09-05 19:08:07.899421: Current learning rate: 0.00838 
+2024-09-05 19:13:16.986581: train_loss -0.7166 
+2024-09-05 19:13:16.994135: val_loss -0.7252 
+2024-09-05 19:13:16.995075: Pseudo dice [0.9579, 0.9507, 0.9751, 0.9389, 0.8474, 0.955, 0.9892] 
+2024-09-05 19:13:16.995995: Epoch time: 309.1 s 
+2024-09-05 19:13:19.860654:  
+2024-09-05 19:13:19.863173: Epoch 179 
+2024-09-05 19:13:19.863966: Current learning rate: 0.00837 
+2024-09-05 19:18:18.341134: train_loss -0.7059 
+2024-09-05 19:18:18.343864: val_loss -0.7053 
+2024-09-05 19:18:18.344598: Pseudo dice [0.9556, 0.9513, 0.9766, 0.9349, 0.8407, 0.948, 0.9888] 
+2024-09-05 19:18:18.345456: Epoch time: 298.49 s 
+2024-09-05 19:18:21.275726:  
+2024-09-05 19:18:21.276885: Epoch 180 
+2024-09-05 19:18:21.277759: Current learning rate: 0.00836 
+2024-09-05 19:23:11.084792: train_loss -0.7107 
+2024-09-05 19:23:11.086635: val_loss -0.7389 
+2024-09-05 19:23:11.087657: Pseudo dice [0.9609, 0.9537, 0.9775, 0.9455, 0.8594, 0.9622, 0.9904] 
+2024-09-05 19:23:11.088973: Epoch time: 289.82 s 
+2024-09-05 19:23:13.797527:  
+2024-09-05 19:23:13.798673: Epoch 181 
+2024-09-05 19:23:13.799311: Current learning rate: 0.00836 
+2024-09-05 19:28:03.306115: train_loss -0.7166 
+2024-09-05 19:28:03.310154: val_loss -0.7041 
+2024-09-05 19:28:03.311154: Pseudo dice [0.9536, 0.9468, 0.9753, 0.9434, 0.8499, 0.9587, 0.9852] 
+2024-09-05 19:28:03.311821: Epoch time: 289.51 s 
+2024-09-05 19:28:05.094583:  
+2024-09-05 19:28:05.106780: Epoch 182 
+2024-09-05 19:28:05.129761: Current learning rate: 0.00835 
+2024-09-05 19:33:06.767954: train_loss -0.7196 
+2024-09-05 19:33:06.770571: val_loss -0.7105 
+2024-09-05 19:33:06.771456: Pseudo dice [0.9562, 0.9524, 0.9737, 0.9429, 0.8422, 0.9608, 0.9875] 
+2024-09-05 19:33:06.772255: Epoch time: 301.68 s 
+2024-09-05 19:33:09.328145:  
+2024-09-05 19:33:09.329277: Epoch 183 
+2024-09-05 19:33:09.330016: Current learning rate: 0.00834 
+2024-09-05 19:38:13.818111: train_loss -0.7161 
+2024-09-05 19:38:13.820596: val_loss -0.7249 
+2024-09-05 19:38:13.821902: Pseudo dice [0.9563, 0.9546, 0.9719, 0.9293, 0.843, 0.9395, 0.9894] 
+2024-09-05 19:38:13.822969: Epoch time: 304.5 s 
+2024-09-05 19:38:16.304637:  
+2024-09-05 19:38:16.306035: Epoch 184 
+2024-09-05 19:38:16.306870: Current learning rate: 0.00833 
+2024-09-05 19:43:08.959489: train_loss -0.716 
+2024-09-05 19:43:08.966759: val_loss -0.7393 
+2024-09-05 19:43:08.968566: Pseudo dice [0.9605, 0.9519, 0.9775, 0.9432, 0.8544, 0.9625, 0.9896] 
+2024-09-05 19:43:08.971541: Epoch time: 292.66 s 
+2024-09-05 19:43:11.787981:  
+2024-09-05 19:43:11.802578: Epoch 185 
+2024-09-05 19:43:11.803469: Current learning rate: 0.00832 
+2024-09-05 19:48:04.631490: train_loss -0.7162 
+2024-09-05 19:48:04.648006: val_loss -0.7484 
+2024-09-05 19:48:04.649102: Pseudo dice [0.9618, 0.953, 0.9796, 0.9453, 0.866, 0.9612, 0.9899] 
+2024-09-05 19:48:04.650046: Epoch time: 292.85 s 
+2024-09-05 19:48:08.548029:  
+2024-09-05 19:48:08.549146: Epoch 186 
+2024-09-05 19:48:08.549885: Current learning rate: 0.00831 
+2024-09-05 19:53:19.028008: train_loss -0.7317 
+2024-09-05 19:53:19.031244: val_loss -0.6991 
+2024-09-05 19:53:19.032429: Pseudo dice [0.9619, 0.951, 0.9784, 0.9402, 0.8599, 0.9659, 0.9902] 
+2024-09-05 19:53:19.033651: Epoch time: 310.49 s 
+2024-09-05 19:53:21.686572:  
+2024-09-05 19:53:21.687691: Epoch 187 
+2024-09-05 19:53:21.688583: Current learning rate: 0.0083 
+2024-09-05 19:58:19.506046: train_loss -0.7072 
+2024-09-05 19:58:19.512932: val_loss -0.7106 
+2024-09-05 19:58:19.515827: Pseudo dice [0.9618, 0.9524, 0.9776, 0.9472, 0.863, 0.9644, 0.9881] 
+2024-09-05 19:58:19.518199: Epoch time: 297.83 s 
+2024-09-05 19:58:22.317222:  
+2024-09-05 19:58:22.331334: Epoch 188 
+2024-09-05 19:58:22.333735: Current learning rate: 0.00829 
+2024-09-05 20:02:56.163078: train_loss -0.7244 
+2024-09-05 20:02:56.184625: val_loss -0.7526 
+2024-09-05 20:02:56.186074: Pseudo dice [0.9585, 0.9475, 0.9768, 0.9457, 0.8575, 0.9638, 0.9908] 
+2024-09-05 20:02:56.188075: Epoch time: 273.85 s 
+2024-09-05 20:02:59.312097:  
+2024-09-05 20:02:59.315305: Epoch 189 
+2024-09-05 20:02:59.317619: Current learning rate: 0.00828 
+2024-09-05 20:08:16.031520: train_loss -0.7224 
+2024-09-05 20:08:16.033214: val_loss -0.7202 
+2024-09-05 20:08:16.034182: Pseudo dice [0.9561, 0.9533, 0.973, 0.9367, 0.8472, 0.9504, 0.9867] 
+2024-09-05 20:08:16.034968: Epoch time: 316.73 s 
+2024-09-05 20:08:18.658028:  
+2024-09-05 20:08:18.659136: Epoch 190 
+2024-09-05 20:08:18.659982: Current learning rate: 0.00827 
+2024-09-05 20:13:14.787448: train_loss -0.7066 
+2024-09-05 20:13:14.788872: val_loss -0.7309 
+2024-09-05 20:13:14.789662: Pseudo dice [0.9581, 0.9484, 0.9743, 0.9346, 0.8528, 0.9609, 0.9901] 
+2024-09-05 20:13:14.790435: Epoch time: 296.14 s 
+2024-09-05 20:13:20.406848:  
+2024-09-05 20:13:20.408874: Epoch 191 
+2024-09-05 20:13:20.409744: Current learning rate: 0.00826 
+2024-09-05 20:18:11.034729: train_loss -0.7162 
+2024-09-05 20:18:11.037455: val_loss -0.6978 
+2024-09-05 20:18:11.038411: Pseudo dice [0.9601, 0.9533, 0.97, 0.9422, 0.8616, 0.9608, 0.9886] 
+2024-09-05 20:18:11.039357: Epoch time: 290.63 s 
+2024-09-05 20:18:13.698372:  
+2024-09-05 20:18:13.699559: Epoch 192 
+2024-09-05 20:18:13.700414: Current learning rate: 0.00825 
+2024-09-05 20:22:41.568854: train_loss -0.7175 
+2024-09-05 20:22:41.571423: val_loss -0.7349 
+2024-09-05 20:22:41.572232: Pseudo dice [0.9602, 0.9515, 0.9794, 0.9494, 0.8658, 0.9617, 0.9883] 
+2024-09-05 20:22:41.572966: Epoch time: 267.88 s 
+2024-09-05 20:22:41.573581: Yayy! New best EMA pseudo Dice: 0.9461 
+2024-09-05 20:22:49.669066:  
+2024-09-05 20:22:49.670355: Epoch 193 
+2024-09-05 20:22:49.671318: Current learning rate: 0.00824 
+2024-09-05 20:27:45.617928: train_loss -0.7177 
+2024-09-05 20:27:45.643369: val_loss -0.7194 
+2024-09-05 20:27:45.644147: Pseudo dice [0.9565, 0.953, 0.9694, 0.9398, 0.8533, 0.9572, 0.9886] 
+2024-09-05 20:27:45.644924: Epoch time: 295.95 s 
+2024-09-05 20:27:48.703001:  
+2024-09-05 20:27:48.704185: Epoch 194 
+2024-09-05 20:27:48.705006: Current learning rate: 0.00824 
+2024-09-05 20:32:44.072810: train_loss -0.7184 
+2024-09-05 20:32:44.095464: val_loss -0.7262 
+2024-09-05 20:32:44.096206: Pseudo dice [0.9593, 0.9566, 0.9752, 0.9453, 0.8603, 0.9586, 0.9899] 
+2024-09-05 20:32:44.096908: Epoch time: 295.38 s 
+2024-09-05 20:32:44.097409: Yayy! New best EMA pseudo Dice: 0.9464 
+2024-09-05 20:32:52.997644:  
+2024-09-05 20:32:52.998940: Epoch 195 
+2024-09-05 20:32:53.000013: Current learning rate: 0.00823 
+2024-09-05 20:37:34.965105: train_loss -0.707 
+2024-09-05 20:37:34.980283: val_loss -0.7139 
+2024-09-05 20:37:34.981395: Pseudo dice [0.9552, 0.9496, 0.974, 0.9416, 0.8555, 0.9626, 0.9872] 
+2024-09-05 20:37:34.982595: Epoch time: 281.97 s 
+2024-09-05 20:37:34.983679: Yayy! New best EMA pseudo Dice: 0.9464 
+2024-09-05 20:37:44.225758:  
+2024-09-05 20:37:44.226894: Epoch 196 
+2024-09-05 20:37:44.227674: Current learning rate: 0.00822 
+2024-09-05 20:42:42.959352: train_loss -0.7099 
+2024-09-05 20:42:42.976209: val_loss -0.7244 
+2024-09-05 20:42:42.976979: Pseudo dice [0.9592, 0.9501, 0.976, 0.9417, 0.8597, 0.9638, 0.9898] 
+2024-09-05 20:42:42.978330: Epoch time: 298.74 s 
+2024-09-05 20:42:42.978946: Yayy! New best EMA pseudo Dice: 0.9466 
+2024-09-05 20:42:52.891050:  
+2024-09-05 20:42:52.892356: Epoch 197 
+2024-09-05 20:42:52.893057: Current learning rate: 0.00821 
+2024-09-05 20:47:47.942312: train_loss -0.7054 
+2024-09-05 20:47:47.959987: val_loss -0.6963 
+2024-09-05 20:47:47.960710: Pseudo dice [0.9577, 0.9499, 0.9721, 0.9405, 0.8555, 0.9655, 0.9897] 
+2024-09-05 20:47:47.961491: Epoch time: 295.06 s 
+2024-09-05 20:47:47.962097: Yayy! New best EMA pseudo Dice: 0.9467 
+2024-09-05 20:47:55.651182:  
+2024-09-05 20:47:55.652272: Epoch 198 
+2024-09-05 20:47:55.653009: Current learning rate: 0.0082 
+2024-09-05 20:52:35.850996: train_loss -0.7227 
+2024-09-05 20:52:35.853804: val_loss -0.7324 
+2024-09-05 20:52:35.854800: Pseudo dice [0.9585, 0.9539, 0.977, 0.9459, 0.8569, 0.9601, 0.9898] 
+2024-09-05 20:52:35.855587: Epoch time: 280.2 s 
+2024-09-05 20:52:35.856343: Yayy! New best EMA pseudo Dice: 0.9469 
+2024-09-05 20:52:45.939383:  
+2024-09-05 20:52:45.940439: Epoch 199 
+2024-09-05 20:52:45.941283: Current learning rate: 0.00819 
+2024-09-05 20:57:37.682140: train_loss -0.7131 
+2024-09-05 20:57:37.700161: val_loss -0.7161 
+2024-09-05 20:57:37.700987: Pseudo dice [0.9578, 0.9516, 0.9733, 0.9411, 0.8464, 0.953, 0.9839] 
+2024-09-05 20:57:37.701706: Epoch time: 291.75 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_6_00_30_15.txt

@@ -0,0 +1,377 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-06 00:30:18.748406: Using torch.compile... 
+2024-09-06 00:30:29.672037: do_dummy_2d_data_aug: False 
+2024-09-06 00:30:29.679499: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-06 00:30:29.681705: The split file contains 5 splits. 
+2024-09-06 00:30:29.682391: Desired fold for training: 0 
+2024-09-06 00:30:29.682986: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-06 00:30:43.502952: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-06 00:30:45.356498:  
+2024-09-06 00:30:45.357683: Epoch 200 
+2024-09-06 00:30:45.359113: Current learning rate: 0.00818 
+2024-09-06 00:49:19.709007: train_loss -0.7197 
+2024-09-06 00:49:19.710448: val_loss -0.733 
+2024-09-06 00:49:19.711691: Pseudo dice [0.9567, 0.9472, 0.9738, 0.932, 0.8681, 0.9632, 0.9901] 
+2024-09-06 00:49:19.712334: Epoch time: 1114.36 s 
+2024-09-06 00:49:21.535512:  
+2024-09-06 00:49:21.536278: Epoch 201 
+2024-09-06 00:49:21.536889: Current learning rate: 0.00817 
+2024-09-06 00:54:12.670696: train_loss -0.7038 
+2024-09-06 00:54:12.673242: val_loss -0.7393 
+2024-09-06 00:54:12.677469: Pseudo dice [0.9578, 0.9503, 0.9777, 0.9384, 0.8552, 0.9643, 0.9903] 
+2024-09-06 00:54:12.679554: Epoch time: 291.14 s 
+2024-09-06 00:54:16.312773:  
+2024-09-06 00:54:16.314003: Epoch 202 
+2024-09-06 00:54:16.315553: Current learning rate: 0.00816 
+2024-09-06 00:59:16.935077: train_loss -0.7186 
+2024-09-06 00:59:16.951245: val_loss -0.7484 
+2024-09-06 00:59:16.952184: Pseudo dice [0.9626, 0.9543, 0.9783, 0.9441, 0.861, 0.9609, 0.9901] 
+2024-09-06 00:59:16.953223: Epoch time: 300.63 s 
+2024-09-06 00:59:16.954887: Yayy! New best EMA pseudo Dice: 0.9471 
+2024-09-06 00:59:25.665431:  
+2024-09-06 00:59:25.666492: Epoch 203 
+2024-09-06 00:59:25.667216: Current learning rate: 0.00815 
+2024-09-06 01:04:22.590609: train_loss -0.7265 
+2024-09-06 01:04:22.611345: val_loss -0.7707 
+2024-09-06 01:04:22.612344: Pseudo dice [0.9597, 0.9549, 0.9781, 0.9449, 0.8552, 0.9652, 0.9892] 
+2024-09-06 01:04:22.613173: Epoch time: 296.93 s 
+2024-09-06 01:04:22.613755: Yayy! New best EMA pseudo Dice: 0.9474 
+2024-09-06 01:04:31.702869:  
+2024-09-06 01:04:31.704043: Epoch 204 
+2024-09-06 01:04:31.704952: Current learning rate: 0.00814 
+2024-09-06 01:09:34.823595: train_loss -0.7182 
+2024-09-06 01:09:34.850940: val_loss -0.726 
+2024-09-06 01:09:34.851900: Pseudo dice [0.9605, 0.9535, 0.9756, 0.9417, 0.8644, 0.9618, 0.9899] 
+2024-09-06 01:09:34.853168: Epoch time: 303.13 s 
+2024-09-06 01:09:34.854176: Yayy! New best EMA pseudo Dice: 0.9476 
+2024-09-06 01:09:45.984502:  
+2024-09-06 01:09:45.985740: Epoch 205 
+2024-09-06 01:09:45.986832: Current learning rate: 0.00813 
+2024-09-06 01:14:25.190864: train_loss -0.7277 
+2024-09-06 01:14:25.196058: val_loss -0.721 
+2024-09-06 01:14:25.197195: Pseudo dice [0.9602, 0.9557, 0.9772, 0.9456, 0.8707, 0.9666, 0.9909] 
+2024-09-06 01:14:25.198205: Epoch time: 279.21 s 
+2024-09-06 01:14:25.199691: Yayy! New best EMA pseudo Dice: 0.9481 
+2024-09-06 01:14:33.905180:  
+2024-09-06 01:14:33.906341: Epoch 206 
+2024-09-06 01:14:33.908055: Current learning rate: 0.00813 
+2024-09-06 01:19:08.363782: train_loss -0.7338 
+2024-09-06 01:19:08.380578: val_loss -0.7317 
+2024-09-06 01:19:08.382201: Pseudo dice [0.961, 0.9544, 0.9781, 0.9375, 0.8539, 0.9616, 0.9868] 
+2024-09-06 01:19:08.383540: Epoch time: 274.46 s 
+2024-09-06 01:19:11.438634:  
+2024-09-06 01:19:11.439728: Epoch 207 
+2024-09-06 01:19:11.440758: Current learning rate: 0.00812 
+2024-09-06 01:23:46.682500: train_loss -0.7149 
+2024-09-06 01:23:46.695547: val_loss -0.761 
+2024-09-06 01:23:46.696697: Pseudo dice [0.9635, 0.9551, 0.9775, 0.9416, 0.8663, 0.9649, 0.9907] 
+2024-09-06 01:23:46.697685: Epoch time: 275.25 s 
+2024-09-06 01:23:46.698786: Yayy! New best EMA pseudo Dice: 0.9484 
+2024-09-06 01:23:56.647382:  
+2024-09-06 01:23:56.648497: Epoch 208 
+2024-09-06 01:23:56.650118: Current learning rate: 0.00811 
+2024-09-06 01:28:42.299994: train_loss -0.7252 
+2024-09-06 01:28:42.327152: val_loss -0.717 
+2024-09-06 01:28:42.328504: Pseudo dice [0.9606, 0.9531, 0.9734, 0.9372, 0.8487, 0.9592, 0.9917] 
+2024-09-06 01:28:42.330049: Epoch time: 285.66 s 
+2024-09-06 01:28:46.592069:  
+2024-09-06 01:28:46.593695: Epoch 209 
+2024-09-06 01:28:46.595210: Current learning rate: 0.0081 
+2024-09-06 01:33:34.728768: train_loss -0.7193 
+2024-09-06 01:33:34.743100: val_loss -0.745 
+2024-09-06 01:33:34.744865: Pseudo dice [0.9571, 0.9528, 0.9724, 0.935, 0.844, 0.9449, 0.9897] 
+2024-09-06 01:33:34.746463: Epoch time: 288.14 s 
+2024-09-06 01:33:40.208870:  
+2024-09-06 01:33:40.210091: Epoch 210 
+2024-09-06 01:33:40.210822: Current learning rate: 0.00809 
+2024-09-06 01:38:43.422899: train_loss -0.7174 
+2024-09-06 01:38:43.476975: val_loss -0.7184 
+2024-09-06 01:38:43.479088: Pseudo dice [0.9598, 0.951, 0.9764, 0.9406, 0.8564, 0.9613, 0.991] 
+2024-09-06 01:38:43.480343: Epoch time: 303.22 s 
+2024-09-06 01:38:46.928847:  
+2024-09-06 01:38:46.930384: Epoch 211 
+2024-09-06 01:38:46.931407: Current learning rate: 0.00808 
+2024-09-06 01:43:29.223972: train_loss -0.722 
+2024-09-06 01:43:29.225368: val_loss -0.6892 
+2024-09-06 01:43:29.227592: Pseudo dice [0.9569, 0.9529, 0.9702, 0.9421, 0.8625, 0.962, 0.9861] 
+2024-09-06 01:43:29.229423: Epoch time: 282.3 s 
+2024-09-06 01:43:32.972460:  
+2024-09-06 01:43:32.973629: Epoch 212 
+2024-09-06 01:43:32.975618: Current learning rate: 0.00807 
+2024-09-06 01:48:31.792915: train_loss -0.7225 
+2024-09-06 01:48:31.807120: val_loss -0.6823 
+2024-09-06 01:48:31.808671: Pseudo dice [0.9581, 0.9538, 0.9748, 0.9417, 0.855, 0.9582, 0.9885] 
+2024-09-06 01:48:31.810430: Epoch time: 298.83 s 
+2024-09-06 01:48:36.098278:  
+2024-09-06 01:48:36.255568: Epoch 213 
+2024-09-06 01:48:36.352691: Current learning rate: 0.00806 
+2024-09-06 01:53:45.125396: train_loss -0.7309 
+2024-09-06 01:53:45.139587: val_loss -0.7716 
+2024-09-06 01:53:45.141289: Pseudo dice [0.9612, 0.9522, 0.9781, 0.9438, 0.87, 0.9655, 0.9915] 
+2024-09-06 01:53:45.142573: Epoch time: 309.03 s 
+2024-09-06 01:53:47.747496:  
+2024-09-06 01:53:47.748860: Epoch 214 
+2024-09-06 01:53:47.749963: Current learning rate: 0.00805 
+2024-09-06 01:58:41.096746: train_loss -0.7315 
+2024-09-06 01:58:41.111232: val_loss -0.7046 
+2024-09-06 01:58:41.112204: Pseudo dice [0.9603, 0.9544, 0.974, 0.9436, 0.8421, 0.9626, 0.9884] 
+2024-09-06 01:58:41.113263: Epoch time: 293.36 s 
+2024-09-06 01:58:44.247914:  
+2024-09-06 01:58:44.249413: Epoch 215 
+2024-09-06 01:58:44.250449: Current learning rate: 0.00804 
+2024-09-06 02:03:40.158468: train_loss -0.7405 
+2024-09-06 02:03:40.175660: val_loss -0.752 
+2024-09-06 02:03:40.176786: Pseudo dice [0.9614, 0.9549, 0.9783, 0.9431, 0.8612, 0.9656, 0.9886] 
+2024-09-06 02:03:40.178319: Epoch time: 295.92 s 
+2024-09-06 02:03:43.976397:  
+2024-09-06 02:03:43.977557: Epoch 216 
+2024-09-06 02:03:43.978547: Current learning rate: 0.00803 
+2024-09-06 02:08:27.727669: train_loss -0.747 
+2024-09-06 02:08:27.729919: val_loss -0.7183 
+2024-09-06 02:08:27.731541: Pseudo dice [0.9617, 0.9572, 0.976, 0.9423, 0.862, 0.9638, 0.991] 
+2024-09-06 02:08:27.732474: Epoch time: 283.76 s 
+2024-09-06 02:08:30.704503:  
+2024-09-06 02:08:30.705905: Epoch 217 
+2024-09-06 02:08:30.706978: Current learning rate: 0.00802 
+2024-09-06 02:13:27.771409: train_loss -0.7343 
+2024-09-06 02:13:27.782614: val_loss -0.7073 
+2024-09-06 02:13:27.783693: Pseudo dice [0.9591, 0.9521, 0.9771, 0.9416, 0.8716, 0.9659, 0.9896] 
+2024-09-06 02:13:27.784934: Epoch time: 297.07 s 
+2024-09-06 02:13:27.786247: Yayy! New best EMA pseudo Dice: 0.9486 
+2024-09-06 02:13:36.575148:  
+2024-09-06 02:13:36.576195: Epoch 218 
+2024-09-06 02:13:36.576971: Current learning rate: 0.00801 
+2024-09-06 02:18:04.448159: train_loss -0.7215 
+2024-09-06 02:18:04.482294: val_loss -0.7129 
+2024-09-06 02:18:04.483432: Pseudo dice [0.9524, 0.9461, 0.9647, 0.943, 0.8433, 0.9586, 0.9836] 
+2024-09-06 02:18:04.484507: Epoch time: 267.88 s 
+2024-09-06 02:18:08.683927:  
+2024-09-06 02:18:08.684886: Epoch 219 
+2024-09-06 02:18:08.686263: Current learning rate: 0.00801 
+2024-09-06 02:23:01.208061: train_loss -0.7232 
+2024-09-06 02:23:01.213077: val_loss -0.7058 
+2024-09-06 02:23:01.214100: Pseudo dice [0.9584, 0.9512, 0.9756, 0.9407, 0.8347, 0.9534, 0.9866] 
+2024-09-06 02:23:01.215068: Epoch time: 292.53 s 
+2024-09-06 02:23:04.427767:  
+2024-09-06 02:23:04.442731: Epoch 220 
+2024-09-06 02:23:04.443924: Current learning rate: 0.008 
+2024-09-06 02:28:04.658054: train_loss -0.7306 
+2024-09-06 02:28:04.659988: val_loss -0.767 
+2024-09-06 02:28:04.661466: Pseudo dice [0.9531, 0.9494, 0.9735, 0.9287, 0.8436, 0.9579, 0.9868] 
+2024-09-06 02:28:04.663083: Epoch time: 300.24 s 
+2024-09-06 02:28:07.331052:  
+2024-09-06 02:28:07.332044: Epoch 221 
+2024-09-06 02:28:07.333454: Current learning rate: 0.00799 
+2024-09-06 02:33:06.074684: train_loss -0.7285 
+2024-09-06 02:33:06.095173: val_loss -0.7035 
+2024-09-06 02:33:06.096123: Pseudo dice [0.96, 0.9538, 0.9767, 0.9435, 0.8704, 0.9652, 0.992] 
+2024-09-06 02:33:06.097165: Epoch time: 298.75 s 
+2024-09-06 02:33:08.784328:  
+2024-09-06 02:33:08.785387: Epoch 222 
+2024-09-06 02:33:08.786337: Current learning rate: 0.00798 
+2024-09-06 02:37:48.663088: train_loss -0.7141 
+2024-09-06 02:37:48.682818: val_loss -0.7088 
+2024-09-06 02:37:48.684209: Pseudo dice [0.9564, 0.9524, 0.9715, 0.93, 0.8552, 0.9594, 0.9866] 
+2024-09-06 02:37:48.690876: Epoch time: 279.88 s 
+2024-09-06 02:37:54.499005:  
+2024-09-06 02:37:54.516167: Epoch 223 
+2024-09-06 02:37:54.517303: Current learning rate: 0.00797 
+2024-09-06 02:42:48.628111: train_loss -0.7296 
+2024-09-06 02:42:48.634796: val_loss -0.7522 
+2024-09-06 02:42:48.637048: Pseudo dice [0.9592, 0.9524, 0.9758, 0.941, 0.8634, 0.9584, 0.9897] 
+2024-09-06 02:42:48.639147: Epoch time: 294.13 s 
+2024-09-06 02:42:51.450813:  
+2024-09-06 02:42:51.453963: Epoch 224 
+2024-09-06 02:42:51.456500: Current learning rate: 0.00796 
+2024-09-06 02:47:28.958263: train_loss -0.7189 
+2024-09-06 02:47:28.962323: val_loss -0.7292 
+2024-09-06 02:47:28.963392: Pseudo dice [0.962, 0.9535, 0.98, 0.9442, 0.8649, 0.966, 0.9899] 
+2024-09-06 02:47:28.964371: Epoch time: 277.52 s 
+2024-09-06 02:47:31.409544:  
+2024-09-06 02:47:31.417783: Epoch 225 
+2024-09-06 02:47:31.419326: Current learning rate: 0.00795 
+2024-09-06 02:52:15.611368: train_loss -0.701 
+2024-09-06 02:52:15.630230: val_loss -0.6924 
+2024-09-06 02:52:15.632282: Pseudo dice [0.9542, 0.9508, 0.9653, 0.9333, 0.8425, 0.954, 0.9887] 
+2024-09-06 02:52:15.633504: Epoch time: 284.21 s 
+2024-09-06 02:52:18.359162:  
+2024-09-06 02:52:18.360061: Epoch 226 
+2024-09-06 02:52:18.361582: Current learning rate: 0.00794 
+2024-09-06 02:57:34.823433: train_loss -0.7236 
+2024-09-06 02:57:34.827872: val_loss -0.735 
+2024-09-06 02:57:34.830027: Pseudo dice [0.9557, 0.95, 0.9723, 0.942, 0.8728, 0.9621, 0.9892] 
+2024-09-06 02:57:34.852192: Epoch time: 316.47 s 
+2024-09-06 02:57:37.726950:  
+2024-09-06 02:57:37.728012: Epoch 227 
+2024-09-06 02:57:37.728767: Current learning rate: 0.00793 
+2024-09-06 03:02:49.217053: train_loss -0.7136 
+2024-09-06 03:02:49.223451: val_loss -0.7356 
+2024-09-06 03:02:49.224415: Pseudo dice [0.9621, 0.9528, 0.9747, 0.9394, 0.8642, 0.9637, 0.9902] 
+2024-09-06 03:02:49.225380: Epoch time: 311.5 s 
+2024-09-06 03:02:52.175227:  
+2024-09-06 03:02:52.176430: Epoch 228 
+2024-09-06 03:02:52.177474: Current learning rate: 0.00792 
+2024-09-06 03:07:36.578289: train_loss -0.7106 
+2024-09-06 03:07:36.591661: val_loss -0.7004 
+2024-09-06 03:07:36.592958: Pseudo dice [0.9532, 0.9501, 0.9726, 0.9355, 0.8482, 0.9599, 0.9857] 
+2024-09-06 03:07:36.594723: Epoch time: 284.41 s 
+2024-09-06 03:07:40.619189:  
+2024-09-06 03:07:40.624582: Epoch 229 
+2024-09-06 03:07:40.630245: Current learning rate: 0.00791 
+2024-09-06 03:12:59.078507: train_loss -0.7228 
+2024-09-06 03:12:59.102506: val_loss -0.7147 
+2024-09-06 03:12:59.104485: Pseudo dice [0.9609, 0.9523, 0.9737, 0.9393, 0.8482, 0.9636, 0.9879] 
+2024-09-06 03:12:59.106255: Epoch time: 318.46 s 
+2024-09-06 03:13:01.829627:  
+2024-09-06 03:13:01.831453: Epoch 230 
+2024-09-06 03:13:01.833091: Current learning rate: 0.0079 
+2024-09-06 03:18:08.206165: train_loss -0.7167 
+2024-09-06 03:18:08.221507: val_loss -0.7578 
+2024-09-06 03:18:08.222798: Pseudo dice [0.9592, 0.9512, 0.9774, 0.9445, 0.8571, 0.963, 0.9868] 
+2024-09-06 03:18:08.224926: Epoch time: 306.38 s 
+2024-09-06 03:18:11.530344:  
+2024-09-06 03:18:11.531760: Epoch 231 
+2024-09-06 03:18:11.533835: Current learning rate: 0.00789 
+2024-09-06 03:23:18.466065: train_loss -0.7209 
+2024-09-06 03:23:18.482060: val_loss -0.7237 
+2024-09-06 03:23:18.483339: Pseudo dice [0.9565, 0.9531, 0.9707, 0.9408, 0.8483, 0.9613, 0.9877] 
+2024-09-06 03:23:18.485325: Epoch time: 306.94 s 
+2024-09-06 03:23:22.611795:  
+2024-09-06 03:23:22.612975: Epoch 232 
+2024-09-06 03:23:22.613734: Current learning rate: 0.00789 
+2024-09-06 03:28:31.924875: train_loss -0.7077 
+2024-09-06 03:28:31.939265: val_loss -0.747 
+2024-09-06 03:28:31.940355: Pseudo dice [0.9597, 0.9531, 0.9758, 0.9456, 0.859, 0.964, 0.9912] 
+2024-09-06 03:28:31.941232: Epoch time: 309.32 s 
+2024-09-06 03:28:34.487183:  
+2024-09-06 03:28:34.488476: Epoch 233 
+2024-09-06 03:28:34.490498: Current learning rate: 0.00788 
+2024-09-06 03:33:29.443756: train_loss -0.723 
+2024-09-06 03:33:29.459399: val_loss -0.7141 
+2024-09-06 03:33:29.460609: Pseudo dice [0.9608, 0.9545, 0.9747, 0.9371, 0.8335, 0.9557, 0.9842] 
+2024-09-06 03:33:29.461631: Epoch time: 294.96 s 
+2024-09-06 03:33:31.879740:  
+2024-09-06 03:33:31.882222: Epoch 234 
+2024-09-06 03:33:31.884731: Current learning rate: 0.00787 
+2024-09-06 03:38:07.175757: train_loss -0.7192 
+2024-09-06 03:38:07.191338: val_loss -0.7099 
+2024-09-06 03:38:07.194007: Pseudo dice [0.9595, 0.9535, 0.9732, 0.9395, 0.8703, 0.9694, 0.9914] 
+2024-09-06 03:38:07.195677: Epoch time: 275.3 s 
+2024-09-06 03:38:10.188084:  
+2024-09-06 03:38:10.189098: Epoch 235 
+2024-09-06 03:38:10.191091: Current learning rate: 0.00786 
+2024-09-06 03:43:12.383831: train_loss -0.7122 
+2024-09-06 03:43:12.385934: val_loss -0.734 
+2024-09-06 03:43:12.387510: Pseudo dice [0.9613, 0.9547, 0.9762, 0.9403, 0.8598, 0.968, 0.9903] 
+2024-09-06 03:43:12.389002: Epoch time: 302.2 s 
+2024-09-06 03:43:16.614503:  
+2024-09-06 03:43:16.617566: Epoch 236 
+2024-09-06 03:43:16.619176: Current learning rate: 0.00785 
+2024-09-06 03:48:17.312998: train_loss -0.7405 
+2024-09-06 03:48:17.319325: val_loss -0.7292 
+2024-09-06 03:48:17.320494: Pseudo dice [0.9602, 0.9511, 0.9755, 0.9454, 0.8614, 0.9646, 0.9909] 
+2024-09-06 03:48:17.321475: Epoch time: 300.71 s 
+2024-09-06 03:48:20.016560:  
+2024-09-06 03:48:20.017827: Epoch 237 
+2024-09-06 03:48:20.018931: Current learning rate: 0.00784 
+2024-09-06 03:53:21.942369: train_loss -0.7195 
+2024-09-06 03:53:21.959852: val_loss -0.6972 
+2024-09-06 03:53:21.961850: Pseudo dice [0.9615, 0.9556, 0.9771, 0.9444, 0.87, 0.9622, 0.9903] 
+2024-09-06 03:53:21.963533: Epoch time: 301.93 s 
+2024-09-06 03:53:26.634168:  
+2024-09-06 03:53:26.635419: Epoch 238 
+2024-09-06 03:53:26.636423: Current learning rate: 0.00783 
+2024-09-06 03:57:58.709191: train_loss -0.7193 
+2024-09-06 03:57:58.726895: val_loss -0.7081 
+2024-09-06 03:57:58.728381: Pseudo dice [0.9549, 0.9471, 0.97, 0.9341, 0.8602, 0.962, 0.9871] 
+2024-09-06 03:57:58.729815: Epoch time: 272.08 s 
+2024-09-06 03:58:01.693040:  
+2024-09-06 03:58:01.698633: Epoch 239 
+2024-09-06 03:58:01.700191: Current learning rate: 0.00782 
+2024-09-06 04:03:01.324342: train_loss -0.6985 
+2024-09-06 04:03:01.339963: val_loss -0.6968 
+2024-09-06 04:03:01.341563: Pseudo dice [0.9527, 0.9425, 0.9702, 0.9371, 0.8479, 0.9547, 0.989] 
+2024-09-06 04:03:01.342995: Epoch time: 299.64 s 
+2024-09-06 04:03:05.464072:  
+2024-09-06 04:03:05.465962: Epoch 240 
+2024-09-06 04:03:05.468151: Current learning rate: 0.00781 
+2024-09-06 04:08:20.821044: train_loss -0.6949 
+2024-09-06 04:08:20.836121: val_loss -0.7003 
+2024-09-06 04:08:20.837790: Pseudo dice [0.9547, 0.9469, 0.9675, 0.9291, 0.8406, 0.9584, 0.989] 
+2024-09-06 04:08:20.839037: Epoch time: 315.36 s 
+2024-09-06 04:08:25.238890:  
+2024-09-06 04:08:25.240288: Epoch 241 
+2024-09-06 04:08:25.241644: Current learning rate: 0.0078 
+2024-09-06 04:13:14.862656: train_loss -0.7019 
+2024-09-06 04:13:14.878060: val_loss -0.7327 
+2024-09-06 04:13:14.879256: Pseudo dice [0.9559, 0.9491, 0.9695, 0.9288, 0.8411, 0.9583, 0.9861] 
+2024-09-06 04:13:14.880452: Epoch time: 289.63 s 
+2024-09-06 04:13:18.917195:  
+2024-09-06 04:13:18.918253: Epoch 242 
+2024-09-06 04:13:18.919509: Current learning rate: 0.00779 
+2024-09-06 04:18:15.342429: train_loss -0.7147 
+2024-09-06 04:18:15.366108: val_loss -0.7507 
+2024-09-06 04:18:15.367650: Pseudo dice [0.9593, 0.9511, 0.9751, 0.9418, 0.8555, 0.962, 0.9866] 
+2024-09-06 04:18:15.369013: Epoch time: 296.43 s 
+2024-09-06 04:18:18.731974:  
+2024-09-06 04:18:18.733001: Epoch 243 
+2024-09-06 04:18:18.733729: Current learning rate: 0.00778 
+2024-09-06 04:23:33.314143: train_loss -0.7194 
+2024-09-06 04:23:33.324331: val_loss -0.7062 
+2024-09-06 04:23:33.325703: Pseudo dice [0.9618, 0.9554, 0.978, 0.9435, 0.8561, 0.962, 0.9889] 
+2024-09-06 04:23:33.327491: Epoch time: 314.59 s 
+2024-09-06 04:23:36.664651:  
+2024-09-06 04:23:36.665737: Epoch 244 
+2024-09-06 04:23:36.666528: Current learning rate: 0.00777 
+2024-09-06 04:28:27.404965: train_loss -0.7092 
+2024-09-06 04:28:27.419681: val_loss -0.6862 
+2024-09-06 04:28:27.421154: Pseudo dice [0.9565, 0.9525, 0.9753, 0.9361, 0.83, 0.9621, 0.9877] 
+2024-09-06 04:28:27.422619: Epoch time: 290.75 s 
+2024-09-06 04:28:29.854615:  
+2024-09-06 04:28:29.867332: Epoch 245 
+2024-09-06 04:28:29.868367: Current learning rate: 0.00777 
+2024-09-06 04:33:44.739736: train_loss -0.7166 
+2024-09-06 04:33:44.759472: val_loss -0.7389 
+2024-09-06 04:33:44.760595: Pseudo dice [0.9549, 0.9514, 0.9709, 0.9351, 0.8561, 0.9599, 0.9859] 
+2024-09-06 04:33:44.761728: Epoch time: 314.89 s 
+2024-09-06 04:33:48.027031:  
+2024-09-06 04:33:48.028280: Epoch 246 
+2024-09-06 04:33:48.029373: Current learning rate: 0.00776 
+2024-09-06 04:38:26.994797: train_loss -0.7002 
+2024-09-06 04:38:27.011744: val_loss -0.776 
+2024-09-06 04:38:27.013300: Pseudo dice [0.9609, 0.9506, 0.9724, 0.9429, 0.8544, 0.9617, 0.987] 
+2024-09-06 04:38:27.014641: Epoch time: 278.97 s 
+2024-09-06 04:38:31.344168:  
+2024-09-06 04:38:31.354794: Epoch 247 
+2024-09-06 04:38:31.355917: Current learning rate: 0.00775 
+2024-09-06 04:43:52.012709: train_loss -0.7046 
+2024-09-06 04:43:52.039072: val_loss -0.7327 
+2024-09-06 04:43:52.040153: Pseudo dice [0.9559, 0.953, 0.9729, 0.9393, 0.8486, 0.9578, 0.9877] 
+2024-09-06 04:43:52.041215: Epoch time: 320.67 s 
+2024-09-06 04:43:54.668405:  
+2024-09-06 04:43:54.669653: Epoch 248 
+2024-09-06 04:43:54.670659: Current learning rate: 0.00774 
+2024-09-06 04:48:48.780704: train_loss -0.7347 
+2024-09-06 04:48:48.801455: val_loss -0.7414 
+2024-09-06 04:48:48.802574: Pseudo dice [0.9625, 0.9568, 0.9773, 0.9508, 0.871, 0.9668, 0.9903] 
+2024-09-06 04:48:48.803625: Epoch time: 294.12 s 
+2024-09-06 04:48:51.592553:  
+2024-09-06 04:48:51.593833: Epoch 249 
+2024-09-06 04:48:51.595046: Current learning rate: 0.00773 
+2024-09-06 04:53:47.614362: train_loss -0.7278 
+2024-09-06 04:53:47.631975: val_loss -0.7574 
+2024-09-06 04:53:47.633262: Pseudo dice [0.9613, 0.9554, 0.9767, 0.9456, 0.8764, 0.9662, 0.9911] 
+2024-09-06 04:53:47.634322: Epoch time: 296.03 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_6_08_29_45.txt

@@ -0,0 +1,376 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-06 08:29:48.994122: Using torch.compile... 
+2024-09-06 08:29:57.625815: do_dummy_2d_data_aug: False 
+2024-09-06 08:29:57.632146: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-06 08:29:57.633682: The split file contains 5 splits. 
+2024-09-06 08:29:57.634734: Desired fold for training: 0 
+2024-09-06 08:29:57.635884: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-06 08:30:09.015288: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-06 08:30:10.407605:  
+2024-09-06 08:30:10.408697: Epoch 250 
+2024-09-06 08:30:10.409446: Current learning rate: 0.00772 
+2024-09-06 08:48:45.077104: train_loss -0.7309 
+2024-09-06 08:48:45.078999: val_loss -0.7392 
+2024-09-06 08:48:45.079787: Pseudo dice [0.9574, 0.9516, 0.974, 0.945, 0.8824, 0.9672, 0.9908] 
+2024-09-06 08:48:45.080860: Epoch time: 1114.68 s 
+2024-09-06 08:48:46.551185:  
+2024-09-06 08:48:46.552291: Epoch 251 
+2024-09-06 08:48:46.552903: Current learning rate: 0.00771 
+2024-09-06 08:53:25.064588: train_loss -0.7123 
+2024-09-06 08:53:25.080446: val_loss -0.7484 
+2024-09-06 08:53:25.084228: Pseudo dice [0.96, 0.9531, 0.9771, 0.9453, 0.8528, 0.96, 0.9876] 
+2024-09-06 08:53:25.086016: Epoch time: 278.52 s 
+2024-09-06 08:53:27.628021:  
+2024-09-06 08:53:27.629596: Epoch 252 
+2024-09-06 08:53:27.630670: Current learning rate: 0.0077 
+2024-09-06 08:58:12.139567: train_loss -0.7282 
+2024-09-06 08:58:12.142772: val_loss -0.7422 
+2024-09-06 08:58:12.143601: Pseudo dice [0.9624, 0.9525, 0.9785, 0.9478, 0.8654, 0.9603, 0.9901] 
+2024-09-06 08:58:12.144854: Epoch time: 284.52 s 
+2024-09-06 08:58:14.357054:  
+2024-09-06 08:58:14.359048: Epoch 253 
+2024-09-06 08:58:14.361125: Current learning rate: 0.00769 
+2024-09-06 09:02:43.428458: train_loss -0.7206 
+2024-09-06 09:02:43.438855: val_loss -0.7234 
+2024-09-06 09:02:43.439797: Pseudo dice [0.9577, 0.9511, 0.975, 0.9419, 0.8521, 0.9546, 0.9791] 
+2024-09-06 09:02:43.440676: Epoch time: 269.08 s 
+2024-09-06 09:02:45.944849:  
+2024-09-06 09:02:45.966828: Epoch 254 
+2024-09-06 09:02:45.968289: Current learning rate: 0.00768 
+2024-09-06 09:07:42.874733: train_loss -0.6864 
+2024-09-06 09:07:42.876143: val_loss -0.7068 
+2024-09-06 09:07:42.876965: Pseudo dice [0.9587, 0.9508, 0.9745, 0.9405, 0.8433, 0.9564, 0.9859] 
+2024-09-06 09:07:42.877908: Epoch time: 296.94 s 
+2024-09-06 09:07:45.470359:  
+2024-09-06 09:07:45.471290: Epoch 255 
+2024-09-06 09:07:45.472020: Current learning rate: 0.00767 
+2024-09-06 09:12:30.792439: train_loss -0.6823 
+2024-09-06 09:12:30.803426: val_loss -0.6865 
+2024-09-06 09:12:30.804416: Pseudo dice [0.9533, 0.949, 0.9676, 0.9323, 0.8391, 0.9475, 0.988] 
+2024-09-06 09:12:30.805439: Epoch time: 285.33 s 
+2024-09-06 09:12:33.567458:  
+2024-09-06 09:12:33.568624: Epoch 256 
+2024-09-06 09:12:33.569525: Current learning rate: 0.00766 
+2024-09-06 09:17:11.068534: train_loss -0.71 
+2024-09-06 09:17:11.103504: val_loss -0.6808 
+2024-09-06 09:17:11.104675: Pseudo dice [0.9565, 0.9493, 0.9713, 0.9366, 0.8451, 0.958, 0.99] 
+2024-09-06 09:17:11.105765: Epoch time: 277.51 s 
+2024-09-06 09:17:13.822534:  
+2024-09-06 09:17:13.824080: Epoch 257 
+2024-09-06 09:17:13.825274: Current learning rate: 0.00765 
+2024-09-06 09:22:06.112202: train_loss -0.7203 
+2024-09-06 09:22:06.118097: val_loss -0.7205 
+2024-09-06 09:22:06.121712: Pseudo dice [0.9526, 0.9502, 0.967, 0.9344, 0.8394, 0.9418, 0.9887] 
+2024-09-06 09:22:06.123230: Epoch time: 292.29 s 
+2024-09-06 09:22:08.493690:  
+2024-09-06 09:22:08.494943: Epoch 258 
+2024-09-06 09:22:08.496368: Current learning rate: 0.00764 
+2024-09-06 09:26:49.622883: train_loss -0.6925 
+2024-09-06 09:26:49.625492: val_loss -0.7233 
+2024-09-06 09:26:49.626437: Pseudo dice [0.9585, 0.9532, 0.9727, 0.9424, 0.8695, 0.963, 0.9902] 
+2024-09-06 09:26:49.627050: Epoch time: 281.13 s 
+2024-09-06 09:26:52.523840:  
+2024-09-06 09:26:52.525406: Epoch 259 
+2024-09-06 09:26:52.526544: Current learning rate: 0.00764 
+2024-09-06 09:31:27.917837: train_loss -0.7254 
+2024-09-06 09:31:27.919382: val_loss -0.7384 
+2024-09-06 09:31:27.920216: Pseudo dice [0.9623, 0.9528, 0.9784, 0.9411, 0.8693, 0.9657, 0.9883] 
+2024-09-06 09:31:27.921021: Epoch time: 275.4 s 
+2024-09-06 09:31:30.417217:  
+2024-09-06 09:31:30.418830: Epoch 260 
+2024-09-06 09:31:30.420324: Current learning rate: 0.00763 
+2024-09-06 09:36:19.646469: train_loss -0.7288 
+2024-09-06 09:36:19.665890: val_loss -0.7183 
+2024-09-06 09:36:19.667123: Pseudo dice [0.9617, 0.9533, 0.9765, 0.9455, 0.8574, 0.9627, 0.9834] 
+2024-09-06 09:36:19.668110: Epoch time: 289.23 s 
+2024-09-06 09:36:21.952087:  
+2024-09-06 09:36:21.953436: Epoch 261 
+2024-09-06 09:36:21.954593: Current learning rate: 0.00762 
+2024-09-06 09:41:09.344661: train_loss -0.729 
+2024-09-06 09:41:09.356866: val_loss -0.7292 
+2024-09-06 09:41:09.358010: Pseudo dice [0.961, 0.9563, 0.9752, 0.9405, 0.8537, 0.9667, 0.9891] 
+2024-09-06 09:41:09.359145: Epoch time: 287.4 s 
+2024-09-06 09:41:11.808854:  
+2024-09-06 09:41:11.810191: Epoch 262 
+2024-09-06 09:41:11.811725: Current learning rate: 0.00761 
+2024-09-06 09:45:56.696283: train_loss -0.7238 
+2024-09-06 09:45:56.711065: val_loss -0.7195 
+2024-09-06 09:45:56.712116: Pseudo dice [0.962, 0.9554, 0.9763, 0.9461, 0.8584, 0.966, 0.9893] 
+2024-09-06 09:45:56.713089: Epoch time: 284.89 s 
+2024-09-06 09:46:01.962770:  
+2024-09-06 09:46:01.964093: Epoch 263 
+2024-09-06 09:46:01.965685: Current learning rate: 0.0076 
+2024-09-06 09:50:29.661900: train_loss -0.7247 
+2024-09-06 09:50:29.663431: val_loss -0.7313 
+2024-09-06 09:50:29.665888: Pseudo dice [0.9585, 0.9544, 0.9755, 0.9443, 0.853, 0.9664, 0.9894] 
+2024-09-06 09:50:29.667940: Epoch time: 267.7 s 
+2024-09-06 09:50:32.151922:  
+2024-09-06 09:50:32.153653: Epoch 264 
+2024-09-06 09:50:32.154581: Current learning rate: 0.00759 
+2024-09-06 09:55:28.095882: train_loss -0.7238 
+2024-09-06 09:55:28.172501: val_loss -0.7575 
+2024-09-06 09:55:28.279125: Pseudo dice [0.9623, 0.9529, 0.9802, 0.9491, 0.8448, 0.9605, 0.9881] 
+2024-09-06 09:55:28.359420: Epoch time: 295.95 s 
+2024-09-06 09:55:31.207213:  
+2024-09-06 09:55:31.252320: Epoch 265 
+2024-09-06 09:55:31.264490: Current learning rate: 0.00758 
+2024-09-06 10:00:01.883580: train_loss -0.7382 
+2024-09-06 10:00:01.895177: val_loss -0.7409 
+2024-09-06 10:00:01.896446: Pseudo dice [0.9636, 0.9561, 0.9797, 0.9478, 0.8716, 0.9666, 0.9911] 
+2024-09-06 10:00:01.898204: Epoch time: 270.68 s 
+2024-09-06 10:00:04.337948:  
+2024-09-06 10:00:04.339113: Epoch 266 
+2024-09-06 10:00:04.339895: Current learning rate: 0.00757 
+2024-09-06 10:04:44.704264: train_loss -0.7209 
+2024-09-06 10:04:44.714233: val_loss -0.7193 
+2024-09-06 10:04:44.714933: Pseudo dice [0.9585, 0.956, 0.9745, 0.9423, 0.857, 0.9667, 0.9907] 
+2024-09-06 10:04:44.715626: Epoch time: 280.37 s 
+2024-09-06 10:04:47.378757:  
+2024-09-06 10:04:47.380338: Epoch 267 
+2024-09-06 10:04:47.381978: Current learning rate: 0.00756 
+2024-09-06 10:09:30.516746: train_loss -0.7259 
+2024-09-06 10:09:30.518122: val_loss -0.7544 
+2024-09-06 10:09:30.519482: Pseudo dice [0.9635, 0.9569, 0.9784, 0.9473, 0.8774, 0.9667, 0.9904] 
+2024-09-06 10:09:30.520857: Epoch time: 283.14 s 
+2024-09-06 10:09:30.522125: Yayy! New best EMA pseudo Dice: 0.9489 
+2024-09-06 10:09:38.690483:  
+2024-09-06 10:09:38.692281: Epoch 268 
+2024-09-06 10:09:38.693356: Current learning rate: 0.00755 
+2024-09-06 10:14:10.462499: train_loss -0.7177 
+2024-09-06 10:14:10.478870: val_loss -0.7362 
+2024-09-06 10:14:10.482990: Pseudo dice [0.9611, 0.9539, 0.9758, 0.949, 0.8612, 0.9608, 0.9899] 
+2024-09-06 10:14:10.484320: Epoch time: 271.78 s 
+2024-09-06 10:14:10.485623: Yayy! New best EMA pseudo Dice: 0.9491 
+2024-09-06 10:14:20.810888:  
+2024-09-06 10:14:20.812407: Epoch 269 
+2024-09-06 10:14:20.814531: Current learning rate: 0.00754 
+2024-09-06 10:18:58.003771: train_loss -0.7258 
+2024-09-06 10:18:58.018603: val_loss -0.7392 
+2024-09-06 10:18:58.019859: Pseudo dice [0.9589, 0.9503, 0.9735, 0.9351, 0.8322, 0.964, 0.9889] 
+2024-09-06 10:18:58.021008: Epoch time: 277.2 s 
+2024-09-06 10:19:00.640474:  
+2024-09-06 10:19:00.654775: Epoch 270 
+2024-09-06 10:19:00.656074: Current learning rate: 0.00753 
+2024-09-06 10:23:40.658885: train_loss -0.7145 
+2024-09-06 10:23:40.660455: val_loss -0.7132 
+2024-09-06 10:23:40.661654: Pseudo dice [0.9613, 0.9562, 0.9756, 0.9364, 0.8587, 0.9628, 0.9895] 
+2024-09-06 10:23:40.662546: Epoch time: 280.02 s 
+2024-09-06 10:23:42.941468:  
+2024-09-06 10:23:42.942725: Epoch 271 
+2024-09-06 10:23:42.943799: Current learning rate: 0.00752 
+2024-09-06 10:28:34.030941: train_loss -0.7282 
+2024-09-06 10:28:34.032391: val_loss -0.7018 
+2024-09-06 10:28:34.033112: Pseudo dice [0.9614, 0.9577, 0.9748, 0.9448, 0.862, 0.9644, 0.9893] 
+2024-09-06 10:28:34.033695: Epoch time: 291.1 s 
+2024-09-06 10:28:36.660552:  
+2024-09-06 10:28:36.673987: Epoch 272 
+2024-09-06 10:28:36.674964: Current learning rate: 0.00751 
+2024-09-06 10:33:23.135737: train_loss -0.732 
+2024-09-06 10:33:23.145850: val_loss -0.7284 
+2024-09-06 10:33:23.147203: Pseudo dice [0.9617, 0.9556, 0.9767, 0.9427, 0.8552, 0.9489, 0.9882] 
+2024-09-06 10:33:23.148084: Epoch time: 286.48 s 
+2024-09-06 10:33:26.206274:  
+2024-09-06 10:33:26.207210: Epoch 273 
+2024-09-06 10:33:26.207968: Current learning rate: 0.00751 
+2024-09-06 10:38:15.726804: train_loss -0.72 
+2024-09-06 10:38:15.730450: val_loss -0.7231 
+2024-09-06 10:38:15.731577: Pseudo dice [0.9575, 0.9535, 0.9725, 0.9379, 0.8527, 0.9639, 0.9871] 
+2024-09-06 10:38:15.732403: Epoch time: 289.53 s 
+2024-09-06 10:38:17.984197:  
+2024-09-06 10:38:17.985347: Epoch 274 
+2024-09-06 10:38:17.986183: Current learning rate: 0.0075 
+2024-09-06 10:43:14.269817: train_loss -0.7223 
+2024-09-06 10:43:14.273100: val_loss -0.7169 
+2024-09-06 10:43:14.274970: Pseudo dice [0.9613, 0.9557, 0.9748, 0.9418, 0.8533, 0.9643, 0.989] 
+2024-09-06 10:43:14.275923: Epoch time: 296.29 s 
+2024-09-06 10:43:16.975421:  
+2024-09-06 10:43:16.976673: Epoch 275 
+2024-09-06 10:43:16.977518: Current learning rate: 0.00749 
+2024-09-06 10:48:05.444328: train_loss -0.7301 
+2024-09-06 10:48:05.446144: val_loss -0.7694 
+2024-09-06 10:48:05.447009: Pseudo dice [0.9611, 0.9556, 0.9755, 0.9491, 0.8463, 0.9621, 0.9889] 
+2024-09-06 10:48:05.447808: Epoch time: 288.48 s 
+2024-09-06 10:48:08.003027:  
+2024-09-06 10:48:08.017258: Epoch 276 
+2024-09-06 10:48:08.088838: Current learning rate: 0.00748 
+2024-09-06 10:52:57.179498: train_loss -0.7348 
+2024-09-06 10:52:57.180974: val_loss -0.7223 
+2024-09-06 10:52:57.181875: Pseudo dice [0.9568, 0.952, 0.9731, 0.9323, 0.8387, 0.9538, 0.9881] 
+2024-09-06 10:52:57.182550: Epoch time: 289.18 s 
+2024-09-06 10:52:59.555490:  
+2024-09-06 10:52:59.556467: Epoch 277 
+2024-09-06 10:52:59.557156: Current learning rate: 0.00747 
+2024-09-06 10:57:47.860164: train_loss -0.7338 
+2024-09-06 10:57:47.861593: val_loss -0.6563 
+2024-09-06 10:57:47.862337: Pseudo dice [0.9561, 0.952, 0.9683, 0.9466, 0.8727, 0.9647, 0.9893] 
+2024-09-06 10:57:47.862916: Epoch time: 288.31 s 
+2024-09-06 10:57:50.167647:  
+2024-09-06 10:57:50.168710: Epoch 278 
+2024-09-06 10:57:50.169474: Current learning rate: 0.00746 
+2024-09-06 11:02:52.184298: train_loss -0.6972 
+2024-09-06 11:02:52.199683: val_loss -0.7041 
+2024-09-06 11:02:52.200416: Pseudo dice [0.9537, 0.9495, 0.9712, 0.9326, 0.8433, 0.9521, 0.9887] 
+2024-09-06 11:02:52.201254: Epoch time: 302.02 s 
+2024-09-06 11:02:54.695942:  
+2024-09-06 11:02:54.706518: Epoch 279 
+2024-09-06 11:02:54.707545: Current learning rate: 0.00745 
+2024-09-06 11:07:30.945929: train_loss -0.7034 
+2024-09-06 11:07:30.959983: val_loss -0.7326 
+2024-09-06 11:07:30.960780: Pseudo dice [0.9586, 0.9479, 0.9738, 0.9349, 0.8532, 0.9544, 0.9842] 
+2024-09-06 11:07:30.961565: Epoch time: 276.26 s 
+2024-09-06 11:07:33.292559:  
+2024-09-06 11:07:33.294294: Epoch 280 
+2024-09-06 11:07:33.295611: Current learning rate: 0.00744 
+2024-09-06 11:12:14.119891: train_loss -0.706 
+2024-09-06 11:12:14.121186: val_loss -0.7439 
+2024-09-06 11:12:14.121779: Pseudo dice [0.9609, 0.9539, 0.9761, 0.9417, 0.846, 0.9574, 0.991] 
+2024-09-06 11:12:14.122602: Epoch time: 280.83 s 
+2024-09-06 11:12:16.575278:  
+2024-09-06 11:12:16.576413: Epoch 281 
+2024-09-06 11:12:16.577347: Current learning rate: 0.00743 
+2024-09-06 11:17:06.594928: train_loss -0.7297 
+2024-09-06 11:17:06.612010: val_loss -0.6999 
+2024-09-06 11:17:06.612751: Pseudo dice [0.963, 0.9576, 0.9771, 0.9451, 0.8683, 0.9633, 0.9905] 
+2024-09-06 11:17:06.614867: Epoch time: 290.03 s 
+2024-09-06 11:17:09.073092:  
+2024-09-06 11:17:09.073972: Epoch 282 
+2024-09-06 11:17:09.074598: Current learning rate: 0.00742 
+2024-09-06 11:21:39.051363: train_loss -0.7178 
+2024-09-06 11:21:39.064565: val_loss -0.7213 
+2024-09-06 11:21:39.065686: Pseudo dice [0.9589, 0.9524, 0.976, 0.9402, 0.8582, 0.962, 0.9889] 
+2024-09-06 11:21:39.066565: Epoch time: 269.98 s 
+2024-09-06 11:21:41.760868:  
+2024-09-06 11:21:41.762682: Epoch 283 
+2024-09-06 11:21:41.764105: Current learning rate: 0.00741 
+2024-09-06 11:26:18.210586: train_loss -0.7121 
+2024-09-06 11:26:18.212690: val_loss -0.7334 
+2024-09-06 11:26:18.214006: Pseudo dice [0.9594, 0.9501, 0.9745, 0.9431, 0.8498, 0.9612, 0.9904] 
+2024-09-06 11:26:18.215194: Epoch time: 276.46 s 
+2024-09-06 11:26:20.715587:  
+2024-09-06 11:26:20.716769: Epoch 284 
+2024-09-06 11:26:20.717489: Current learning rate: 0.0074 
+2024-09-06 11:30:58.850335: train_loss -0.7244 
+2024-09-06 11:30:58.863975: val_loss -0.741 
+2024-09-06 11:30:58.865368: Pseudo dice [0.9614, 0.9542, 0.9773, 0.9419, 0.8588, 0.964, 0.99] 
+2024-09-06 11:30:58.866576: Epoch time: 278.14 s 
+2024-09-06 11:31:01.238369:  
+2024-09-06 11:31:01.239411: Epoch 285 
+2024-09-06 11:31:01.240242: Current learning rate: 0.00739 
+2024-09-06 11:35:49.187400: train_loss -0.7263 
+2024-09-06 11:35:49.189435: val_loss -0.7403 
+2024-09-06 11:35:49.194974: Pseudo dice [0.9622, 0.9533, 0.9774, 0.9411, 0.8652, 0.968, 0.9899] 
+2024-09-06 11:35:49.196062: Epoch time: 287.95 s 
+2024-09-06 11:35:51.864773:  
+2024-09-06 11:35:51.867490: Epoch 286 
+2024-09-06 11:35:51.868260: Current learning rate: 0.00738 
+2024-09-06 11:40:51.074928: train_loss -0.7529 
+2024-09-06 11:40:51.085083: val_loss -0.7293 
+2024-09-06 11:40:51.085893: Pseudo dice [0.9576, 0.9524, 0.9758, 0.9364, 0.8396, 0.9622, 0.9883] 
+2024-09-06 11:40:51.086719: Epoch time: 299.22 s 
+2024-09-06 11:40:54.051197:  
+2024-09-06 11:40:54.061141: Epoch 287 
+2024-09-06 11:40:54.062051: Current learning rate: 0.00738 
+2024-09-06 11:45:37.192760: train_loss -0.7371 
+2024-09-06 11:45:37.195271: val_loss -0.7312 
+2024-09-06 11:45:37.196125: Pseudo dice [0.96, 0.9567, 0.9745, 0.9454, 0.856, 0.9678, 0.9915] 
+2024-09-06 11:45:37.196910: Epoch time: 283.15 s 
+2024-09-06 11:45:40.023453:  
+2024-09-06 11:45:40.034708: Epoch 288 
+2024-09-06 11:45:40.035694: Current learning rate: 0.00737 
+2024-09-06 11:50:18.340444: train_loss -0.7251 
+2024-09-06 11:50:18.343143: val_loss -0.7336 
+2024-09-06 11:50:18.343830: Pseudo dice [0.9602, 0.9534, 0.9756, 0.9441, 0.8521, 0.9615, 0.9888] 
+2024-09-06 11:50:18.344478: Epoch time: 278.32 s 
+2024-09-06 11:50:19.834654:  
+2024-09-06 11:50:19.835576: Epoch 289 
+2024-09-06 11:50:19.836403: Current learning rate: 0.00736 
+2024-09-06 11:55:07.388682: train_loss -0.7038 
+2024-09-06 11:55:07.392196: val_loss -0.7423 
+2024-09-06 11:55:07.393341: Pseudo dice [0.9588, 0.9499, 0.9746, 0.9458, 0.8558, 0.9568, 0.9895] 
+2024-09-06 11:55:07.397889: Epoch time: 287.56 s 
+2024-09-06 11:55:10.287790:  
+2024-09-06 11:55:10.288855: Epoch 290 
+2024-09-06 11:55:10.289639: Current learning rate: 0.00735 
+2024-09-06 11:59:51.952800: train_loss -0.7338 
+2024-09-06 11:59:51.955139: val_loss -0.713 
+2024-09-06 11:59:51.955858: Pseudo dice [0.9596, 0.9547, 0.9766, 0.9397, 0.8701, 0.964, 0.99] 
+2024-09-06 11:59:51.960043: Epoch time: 281.67 s 
+2024-09-06 11:59:54.681270:  
+2024-09-06 11:59:54.682394: Epoch 291 
+2024-09-06 11:59:54.683363: Current learning rate: 0.00734 
+2024-09-06 12:04:42.300007: train_loss -0.739 
+2024-09-06 12:04:42.313241: val_loss -0.7485 
+2024-09-06 12:04:42.314497: Pseudo dice [0.9622, 0.9549, 0.9795, 0.9422, 0.8758, 0.9664, 0.9906] 
+2024-09-06 12:04:42.315140: Epoch time: 287.62 s 
+2024-09-06 12:04:44.985881:  
+2024-09-06 12:04:44.987147: Epoch 292 
+2024-09-06 12:04:44.988369: Current learning rate: 0.00733 
+2024-09-06 12:09:22.330969: train_loss -0.7263 
+2024-09-06 12:09:22.347856: val_loss -0.7132 
+2024-09-06 12:09:22.348575: Pseudo dice [0.9611, 0.9553, 0.9801, 0.9462, 0.8638, 0.9653, 0.9899] 
+2024-09-06 12:09:22.349232: Epoch time: 277.35 s 
+2024-09-06 12:09:25.551800:  
+2024-09-06 12:09:25.552817: Epoch 293 
+2024-09-06 12:09:25.553547: Current learning rate: 0.00732 
+2024-09-06 12:14:20.282400: train_loss -0.7232 
+2024-09-06 12:14:20.285016: val_loss -0.7504 
+2024-09-06 12:14:20.285948: Pseudo dice [0.9601, 0.9541, 0.9782, 0.9402, 0.8613, 0.9625, 0.99] 
+2024-09-06 12:14:20.286967: Epoch time: 294.74 s 
+2024-09-06 12:14:22.377725:  
+2024-09-06 12:14:22.378789: Epoch 294 
+2024-09-06 12:14:22.379566: Current learning rate: 0.00731 
+2024-09-06 12:19:14.386244: train_loss -0.7287 
+2024-09-06 12:19:14.388952: val_loss -0.739 
+2024-09-06 12:19:14.389970: Pseudo dice [0.9609, 0.953, 0.9757, 0.9385, 0.864, 0.9681, 0.9903] 
+2024-09-06 12:19:14.390919: Epoch time: 292.01 s 
+2024-09-06 12:19:14.391644: Yayy! New best EMA pseudo Dice: 0.9491 
+2024-09-06 12:19:23.794246:  
+2024-09-06 12:19:23.795141: Epoch 295 
+2024-09-06 12:19:23.795801: Current learning rate: 0.0073 
+2024-09-06 12:24:14.551243: train_loss -0.7258 
+2024-09-06 12:24:14.555010: val_loss -0.7242 
+2024-09-06 12:24:14.556180: Pseudo dice [0.9592, 0.9547, 0.9715, 0.9418, 0.851, 0.9644, 0.9885] 
+2024-09-06 12:24:14.557477: Epoch time: 290.76 s 
+2024-09-06 12:24:17.775421:  
+2024-09-06 12:24:17.776398: Epoch 296 
+2024-09-06 12:24:17.777110: Current learning rate: 0.00729 
+2024-09-06 12:28:56.693326: train_loss -0.715 
+2024-09-06 12:28:56.695658: val_loss -0.7527 
+2024-09-06 12:28:56.696609: Pseudo dice [0.9608, 0.9561, 0.9757, 0.939, 0.8606, 0.9668, 0.9882] 
+2024-09-06 12:28:56.697395: Epoch time: 278.92 s 
+2024-09-06 12:28:59.296468:  
+2024-09-06 12:28:59.297447: Epoch 297 
+2024-09-06 12:28:59.298124: Current learning rate: 0.00728 
+2024-09-06 12:33:24.848214: train_loss -0.7275 
+2024-09-06 12:33:24.849643: val_loss -0.6978 
+2024-09-06 12:33:24.850316: Pseudo dice [0.9622, 0.961, 0.9778, 0.9423, 0.8795, 0.9653, 0.9917] 
+2024-09-06 12:33:24.851068: Epoch time: 265.56 s 
+2024-09-06 12:33:24.851774: Yayy! New best EMA pseudo Dice: 0.9495 
+2024-09-06 12:33:33.109207:  
+2024-09-06 12:33:33.110083: Epoch 298 
+2024-09-06 12:33:33.110802: Current learning rate: 0.00727 
+2024-09-06 12:38:09.914125: train_loss -0.7291 
+2024-09-06 12:38:09.930555: val_loss -0.753 
+2024-09-06 12:38:09.931412: Pseudo dice [0.9616, 0.955, 0.9773, 0.9451, 0.8623, 0.9675, 0.9901] 
+2024-09-06 12:38:09.932142: Epoch time: 276.81 s 
+2024-09-06 12:38:09.932884: Yayy! New best EMA pseudo Dice: 0.9497 
+2024-09-06 12:38:18.487256:  
+2024-09-06 12:38:18.488249: Epoch 299 
+2024-09-06 12:38:18.489026: Current learning rate: 0.00726 
+2024-09-06 12:42:51.125676: train_loss -0.7318 
+2024-09-06 12:42:51.139068: val_loss -0.7435 
+2024-09-06 12:42:51.139867: Pseudo dice [0.9634, 0.9552, 0.9802, 0.9419, 0.8501, 0.9666, 0.9862] 
+2024-09-06 12:42:51.140481: Epoch time: 272.64 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_7_14_05_16.txt

@@ -0,0 +1,381 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-07 14:05:20.211550: Using torch.compile... 
+2024-09-07 14:05:26.355340: do_dummy_2d_data_aug: False 
+2024-09-07 14:05:26.360875: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-07 14:05:26.361969: The split file contains 5 splits. 
+2024-09-07 14:05:26.362309: Desired fold for training: 0 
+2024-09-07 14:05:26.362671: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-07 14:05:37.234429: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-07 14:05:38.169745:  
+2024-09-07 14:05:38.180290: Epoch 300 
+2024-09-07 14:05:38.181097: Current learning rate: 0.00725 
+2024-09-07 14:21:06.131748: train_loss -0.7185 
+2024-09-07 14:21:06.133882: val_loss -0.7748 
+2024-09-07 14:21:06.134394: Pseudo dice [0.9629, 0.9543, 0.9769, 0.9452, 0.8606, 0.966, 0.9908] 
+2024-09-07 14:21:06.134853: Epoch time: 927.97 s 
+2024-09-07 14:21:06.135258: Yayy! New best EMA pseudo Dice: 0.9498 
+2024-09-07 14:21:09.365612:  
+2024-09-07 14:21:09.366222: Epoch 301 
+2024-09-07 14:21:09.366655: Current learning rate: 0.00724 
+2024-09-07 14:25:56.813189: train_loss -0.7423 
+2024-09-07 14:25:56.831361: val_loss -0.7112 
+2024-09-07 14:25:56.832139: Pseudo dice [0.964, 0.9545, 0.9758, 0.9432, 0.8693, 0.9665, 0.9889] 
+2024-09-07 14:25:56.832676: Epoch time: 287.45 s 
+2024-09-07 14:25:56.835197: Yayy! New best EMA pseudo Dice: 0.95 
+2024-09-07 14:26:03.225982:  
+2024-09-07 14:26:03.226821: Epoch 302 
+2024-09-07 14:26:03.227569: Current learning rate: 0.00724 
+2024-09-07 14:31:13.508010: train_loss -0.7356 
+2024-09-07 14:31:13.515500: val_loss -0.739 
+2024-09-07 14:31:13.517344: Pseudo dice [0.9638, 0.9554, 0.9793, 0.9409, 0.8487, 0.9683, 0.9883] 
+2024-09-07 14:31:13.518197: Epoch time: 310.27 s 
+2024-09-07 14:31:16.800194:  
+2024-09-07 14:31:16.810489: Epoch 303 
+2024-09-07 14:31:16.812279: Current learning rate: 0.00723 
+2024-09-07 14:36:15.654032: train_loss -0.7307 
+2024-09-07 14:36:15.666058: val_loss -0.7301 
+2024-09-07 14:36:15.666713: Pseudo dice [0.9586, 0.9527, 0.9764, 0.9402, 0.8682, 0.9661, 0.9896] 
+2024-09-07 14:36:15.667223: Epoch time: 298.86 s 
+2024-09-07 14:36:19.585510:  
+2024-09-07 14:36:19.586416: Epoch 304 
+2024-09-07 14:36:19.586941: Current learning rate: 0.00722 
+2024-09-07 14:41:11.860701: train_loss -0.7326 
+2024-09-07 14:41:11.877699: val_loss -0.7474 
+2024-09-07 14:41:11.878488: Pseudo dice [0.9604, 0.9509, 0.9749, 0.9456, 0.8667, 0.9676, 0.989] 
+2024-09-07 14:41:11.879152: Epoch time: 292.28 s 
+2024-09-07 14:41:11.880046: Yayy! New best EMA pseudo Dice: 0.95 
+2024-09-07 14:41:18.735421:  
+2024-09-07 14:41:18.736454: Epoch 305 
+2024-09-07 14:41:18.737009: Current learning rate: 0.00721 
+2024-09-07 14:47:00.103184: train_loss -0.7253 
+2024-09-07 14:47:00.119946: val_loss -0.7255 
+2024-09-07 14:47:00.120981: Pseudo dice [0.9618, 0.957, 0.9729, 0.9469, 0.87, 0.964, 0.9917] 
+2024-09-07 14:47:00.121698: Epoch time: 341.37 s 
+2024-09-07 14:47:00.122293: Yayy! New best EMA pseudo Dice: 0.9502 
+2024-09-07 14:47:06.525476:  
+2024-09-07 14:47:06.526784: Epoch 306 
+2024-09-07 14:47:06.527516: Current learning rate: 0.0072 
+2024-09-07 14:53:10.953646: train_loss -0.7372 
+2024-09-07 14:53:10.957335: val_loss -0.7583 
+2024-09-07 14:53:10.957885: Pseudo dice [0.9613, 0.9545, 0.9767, 0.9428, 0.8657, 0.966, 0.9886] 
+2024-09-07 14:53:10.958390: Epoch time: 364.44 s 
+2024-09-07 14:53:10.958806: Yayy! New best EMA pseudo Dice: 0.9503 
+2024-09-07 14:53:17.428514:  
+2024-09-07 14:53:17.429941: Epoch 307 
+2024-09-07 14:53:17.430575: Current learning rate: 0.00719 
+2024-09-07 14:58:24.560125: train_loss -0.7394 
+2024-09-07 14:58:24.568820: val_loss -0.7651 
+2024-09-07 14:58:24.569927: Pseudo dice [0.9604, 0.9527, 0.979, 0.9428, 0.8671, 0.9634, 0.9892] 
+2024-09-07 14:58:24.570935: Epoch time: 307.14 s 
+2024-09-07 14:58:24.571971: Yayy! New best EMA pseudo Dice: 0.9503 
+2024-09-07 14:58:33.835978:  
+2024-09-07 14:58:33.837277: Epoch 308 
+2024-09-07 14:58:33.838047: Current learning rate: 0.00718 
+2024-09-07 15:04:05.191993: train_loss -0.7097 
+2024-09-07 15:04:05.207944: val_loss -0.7448 
+2024-09-07 15:04:05.208855: Pseudo dice [0.9604, 0.9541, 0.9769, 0.9447, 0.8665, 0.9666, 0.9899] 
+2024-09-07 15:04:05.209736: Epoch time: 331.36 s 
+2024-09-07 15:04:05.210654: Yayy! New best EMA pseudo Dice: 0.9504 
+2024-09-07 15:04:11.876983:  
+2024-09-07 15:04:11.888408: Epoch 309 
+2024-09-07 15:04:11.890238: Current learning rate: 0.00717 
+2024-09-07 15:09:15.504750: train_loss -0.709 
+2024-09-07 15:09:15.522550: val_loss -0.7462 
+2024-09-07 15:09:15.523728: Pseudo dice [0.9613, 0.9561, 0.9765, 0.9382, 0.8543, 0.9638, 0.9887] 
+2024-09-07 15:09:15.524667: Epoch time: 303.63 s 
+2024-09-07 15:09:18.866999:  
+2024-09-07 15:09:18.888518: Epoch 310 
+2024-09-07 15:09:18.889418: Current learning rate: 0.00716 
+2024-09-07 15:14:53.530306: train_loss -0.7318 
+2024-09-07 15:14:53.539969: val_loss -0.7597 
+2024-09-07 15:14:53.540622: Pseudo dice [0.9634, 0.9554, 0.9787, 0.9423, 0.8697, 0.9667, 0.9919] 
+2024-09-07 15:14:53.541312: Epoch time: 334.67 s 
+2024-09-07 15:14:53.541790: Yayy! New best EMA pseudo Dice: 0.9504 
+2024-09-07 15:14:58.520967:  
+2024-09-07 15:14:58.521922: Epoch 311 
+2024-09-07 15:14:58.522592: Current learning rate: 0.00715 
+2024-09-07 15:19:45.103633: train_loss -0.7157 
+2024-09-07 15:19:45.111042: val_loss -0.7308 
+2024-09-07 15:19:45.111658: Pseudo dice [0.952, 0.952, 0.9705, 0.9351, 0.8561, 0.9626, 0.9884] 
+2024-09-07 15:19:45.112583: Epoch time: 286.59 s 
+2024-09-07 15:19:49.409302:  
+2024-09-07 15:19:49.410480: Epoch 312 
+2024-09-07 15:19:49.411468: Current learning rate: 0.00714 
+2024-09-07 15:24:46.767203: train_loss -0.7184 
+2024-09-07 15:24:46.769648: val_loss -0.7047 
+2024-09-07 15:24:46.770441: Pseudo dice [0.9589, 0.9538, 0.976, 0.9346, 0.8551, 0.9636, 0.9905] 
+2024-09-07 15:24:46.770977: Epoch time: 297.36 s 
+2024-09-07 15:24:50.928598:  
+2024-09-07 15:24:50.940536: Epoch 313 
+2024-09-07 15:24:50.941383: Current learning rate: 0.00713 
+2024-09-07 15:29:47.518383: train_loss -0.7084 
+2024-09-07 15:29:47.539891: val_loss -0.6927 
+2024-09-07 15:29:47.540832: Pseudo dice [0.9517, 0.9466, 0.9677, 0.9215, 0.8298, 0.9514, 0.988] 
+2024-09-07 15:29:47.541476: Epoch time: 296.59 s 
+2024-09-07 15:29:50.333994:  
+2024-09-07 15:29:50.335002: Epoch 314 
+2024-09-07 15:29:50.335717: Current learning rate: 0.00712 
+2024-09-07 15:34:48.038609: train_loss -0.7024 
+2024-09-07 15:34:48.070246: val_loss -0.6764 
+2024-09-07 15:34:48.071455: Pseudo dice [0.9547, 0.9519, 0.9647, 0.9335, 0.8499, 0.9604, 0.9882] 
+2024-09-07 15:34:48.072145: Epoch time: 297.71 s 
+2024-09-07 15:34:51.793449:  
+2024-09-07 15:34:51.794598: Epoch 315 
+2024-09-07 15:34:51.795295: Current learning rate: 0.00711 
+2024-09-07 15:39:58.278038: train_loss -0.7298 
+2024-09-07 15:39:58.280163: val_loss -0.6892 
+2024-09-07 15:39:58.280854: Pseudo dice [0.9552, 0.9512, 0.9689, 0.9381, 0.8609, 0.9586, 0.989] 
+2024-09-07 15:39:58.281337: Epoch time: 306.49 s 
+2024-09-07 15:40:01.840103:  
+2024-09-07 15:40:01.855578: Epoch 316 
+2024-09-07 15:40:01.856363: Current learning rate: 0.0071 
+2024-09-07 15:44:57.414978: train_loss -0.7298 
+2024-09-07 15:44:57.433329: val_loss -0.7565 
+2024-09-07 15:44:57.433932: Pseudo dice [0.9586, 0.9505, 0.9757, 0.9394, 0.8631, 0.9652, 0.9898] 
+2024-09-07 15:44:57.434442: Epoch time: 295.58 s 
+2024-09-07 15:45:01.346862:  
+2024-09-07 15:45:01.363747: Epoch 317 
+2024-09-07 15:45:01.364573: Current learning rate: 0.0071 
+2024-09-07 15:50:07.575720: train_loss -0.7245 
+2024-09-07 15:50:07.589530: val_loss -0.7397 
+2024-09-07 15:50:07.590140: Pseudo dice [0.9577, 0.9517, 0.9765, 0.9373, 0.8528, 0.9586, 0.9885] 
+2024-09-07 15:50:07.590635: Epoch time: 306.23 s 
+2024-09-07 15:50:11.655424:  
+2024-09-07 15:50:11.668409: Epoch 318 
+2024-09-07 15:50:11.669251: Current learning rate: 0.00709 
+2024-09-07 15:55:10.637693: train_loss -0.7127 
+2024-09-07 15:55:10.660609: val_loss -0.7116 
+2024-09-07 15:55:10.664783: Pseudo dice [0.9595, 0.9526, 0.9778, 0.9425, 0.8539, 0.9575, 0.9875] 
+2024-09-07 15:55:10.669531: Epoch time: 298.99 s 
+2024-09-07 15:55:13.806468:  
+2024-09-07 15:55:13.820531: Epoch 319 
+2024-09-07 15:55:13.822236: Current learning rate: 0.00708 
+2024-09-07 16:00:14.988022: train_loss -0.7268 
+2024-09-07 16:00:14.990448: val_loss -0.7517 
+2024-09-07 16:00:14.991291: Pseudo dice [0.9607, 0.9543, 0.9761, 0.9394, 0.8602, 0.9625, 0.9905] 
+2024-09-07 16:00:14.991992: Epoch time: 301.19 s 
+2024-09-07 16:00:19.202133:  
+2024-09-07 16:00:19.218832: Epoch 320 
+2024-09-07 16:00:19.221745: Current learning rate: 0.00707 
+2024-09-07 16:04:59.690048: train_loss -0.7351 
+2024-09-07 16:04:59.701367: val_loss -0.7663 
+2024-09-07 16:04:59.701941: Pseudo dice [0.9621, 0.9546, 0.9788, 0.9507, 0.872, 0.962, 0.9917] 
+2024-09-07 16:04:59.702421: Epoch time: 280.49 s 
+2024-09-07 16:05:05.281230:  
+2024-09-07 16:05:05.296343: Epoch 321 
+2024-09-07 16:05:05.297038: Current learning rate: 0.00706 
+2024-09-07 16:10:02.005168: train_loss -0.7232 
+2024-09-07 16:10:02.020911: val_loss -0.7235 
+2024-09-07 16:10:02.022459: Pseudo dice [0.9608, 0.958, 0.9758, 0.9435, 0.8678, 0.9652, 0.9908] 
+2024-09-07 16:10:02.024105: Epoch time: 296.73 s 
+2024-09-07 16:10:05.514442:  
+2024-09-07 16:10:05.515237: Epoch 322 
+2024-09-07 16:10:05.515829: Current learning rate: 0.00705 
+2024-09-07 16:14:57.809315: train_loss -0.7353 
+2024-09-07 16:14:57.816815: val_loss -0.74 
+2024-09-07 16:14:57.817441: Pseudo dice [0.96, 0.9528, 0.976, 0.9428, 0.8681, 0.9666, 0.989] 
+2024-09-07 16:14:57.818413: Epoch time: 292.3 s 
+2024-09-07 16:15:01.155203:  
+2024-09-07 16:15:01.156909: Epoch 323 
+2024-09-07 16:15:01.158196: Current learning rate: 0.00704 
+2024-09-07 16:19:51.425664: train_loss -0.7467 
+2024-09-07 16:19:51.434794: val_loss -0.7223 
+2024-09-07 16:19:51.435388: Pseudo dice [0.9626, 0.9552, 0.9786, 0.9503, 0.8778, 0.9685, 0.9904] 
+2024-09-07 16:19:51.436045: Epoch time: 290.29 s 
+2024-09-07 16:19:54.683493:  
+2024-09-07 16:19:54.684294: Epoch 324 
+2024-09-07 16:19:54.684914: Current learning rate: 0.00703 
+2024-09-07 16:24:47.870667: train_loss -0.7139 
+2024-09-07 16:24:47.878586: val_loss -0.7515 
+2024-09-07 16:24:47.879161: Pseudo dice [0.9601, 0.954, 0.9756, 0.9491, 0.8497, 0.9606, 0.9861] 
+2024-09-07 16:24:47.880057: Epoch time: 293.19 s 
+2024-09-07 16:24:51.351400:  
+2024-09-07 16:24:51.364268: Epoch 325 
+2024-09-07 16:24:51.365142: Current learning rate: 0.00702 
+2024-09-07 16:29:36.766147: train_loss -0.7208 
+2024-09-07 16:29:36.787106: val_loss -0.7181 
+2024-09-07 16:29:36.787991: Pseudo dice [0.9611, 0.9575, 0.9791, 0.9411, 0.8585, 0.9628, 0.9885] 
+2024-09-07 16:29:36.788597: Epoch time: 285.42 s 
+2024-09-07 16:29:41.038020:  
+2024-09-07 16:29:41.038833: Epoch 326 
+2024-09-07 16:29:41.039712: Current learning rate: 0.00701 
+2024-09-07 16:34:36.655526: train_loss -0.7318 
+2024-09-07 16:34:36.668836: val_loss -0.6933 
+2024-09-07 16:34:36.669505: Pseudo dice [0.9598, 0.9535, 0.973, 0.9431, 0.8518, 0.9482, 0.9875] 
+2024-09-07 16:34:36.670385: Epoch time: 295.62 s 
+2024-09-07 16:34:40.657646:  
+2024-09-07 16:34:40.658860: Epoch 327 
+2024-09-07 16:34:40.659608: Current learning rate: 0.007 
+2024-09-07 16:39:14.438781: train_loss -0.7279 
+2024-09-07 16:39:14.453470: val_loss -0.7467 
+2024-09-07 16:39:14.454147: Pseudo dice [0.9605, 0.9539, 0.9777, 0.9483, 0.8675, 0.9664, 0.9896] 
+2024-09-07 16:39:14.454621: Epoch time: 273.79 s 
+2024-09-07 16:39:19.501521:  
+2024-09-07 16:39:19.502612: Epoch 328 
+2024-09-07 16:39:19.503579: Current learning rate: 0.00699 
+2024-09-07 16:44:08.699061: train_loss -0.7294 
+2024-09-07 16:44:08.714282: val_loss -0.7179 
+2024-09-07 16:44:08.715529: Pseudo dice [0.9613, 0.9532, 0.9761, 0.944, 0.8569, 0.9655, 0.9902] 
+2024-09-07 16:44:08.716230: Epoch time: 289.23 s 
+2024-09-07 16:44:13.349552:  
+2024-09-07 16:44:13.364552: Epoch 329 
+2024-09-07 16:44:13.365245: Current learning rate: 0.00698 
+2024-09-07 16:48:46.150386: train_loss -0.7221 
+2024-09-07 16:48:46.151761: val_loss -0.7572 
+2024-09-07 16:48:46.152644: Pseudo dice [0.9616, 0.9562, 0.9767, 0.9493, 0.8718, 0.9661, 0.9908] 
+2024-09-07 16:48:46.153157: Epoch time: 272.81 s 
+2024-09-07 16:48:48.524622:  
+2024-09-07 16:48:48.526134: Epoch 330 
+2024-09-07 16:48:48.527404: Current learning rate: 0.00697 
+2024-09-07 16:53:26.500555: train_loss -0.723 
+2024-09-07 16:53:26.503242: val_loss -0.6895 
+2024-09-07 16:53:26.504159: Pseudo dice [0.9631, 0.9543, 0.9774, 0.9508, 0.8717, 0.9662, 0.9922] 
+2024-09-07 16:53:26.504861: Epoch time: 277.98 s 
+2024-09-07 16:53:29.138929:  
+2024-09-07 16:53:29.139846: Epoch 331 
+2024-09-07 16:53:29.140493: Current learning rate: 0.00696 
+2024-09-07 16:58:22.131825: train_loss -0.7348 
+2024-09-07 16:58:22.134392: val_loss -0.7324 
+2024-09-07 16:58:22.134932: Pseudo dice [0.9614, 0.9537, 0.9788, 0.9466, 0.8643, 0.9702, 0.991] 
+2024-09-07 16:58:22.135427: Epoch time: 293.0 s 
+2024-09-07 16:58:23.906757:  
+2024-09-07 16:58:23.907948: Epoch 332 
+2024-09-07 16:58:23.908987: Current learning rate: 0.00696 
+2024-09-07 17:03:17.475259: train_loss -0.7392 
+2024-09-07 17:03:17.488908: val_loss -0.715 
+2024-09-07 17:03:17.489629: Pseudo dice [0.9636, 0.9555, 0.9758, 0.9436, 0.856, 0.9669, 0.9892] 
+2024-09-07 17:03:17.490151: Epoch time: 293.57 s 
+2024-09-07 17:03:19.004263:  
+2024-09-07 17:03:19.004913: Epoch 333 
+2024-09-07 17:03:19.005412: Current learning rate: 0.00695 
+2024-09-07 17:08:14.388670: train_loss -0.7391 
+2024-09-07 17:08:14.390670: val_loss -0.7567 
+2024-09-07 17:08:14.391225: Pseudo dice [0.9566, 0.9546, 0.9718, 0.9448, 0.8634, 0.9648, 0.9896] 
+2024-09-07 17:08:14.391709: Epoch time: 295.39 s 
+2024-09-07 17:08:16.728722:  
+2024-09-07 17:08:16.729742: Epoch 334 
+2024-09-07 17:08:16.730396: Current learning rate: 0.00694 
+2024-09-07 17:13:20.321036: train_loss -0.739 
+2024-09-07 17:13:20.329638: val_loss -0.7128 
+2024-09-07 17:13:20.331043: Pseudo dice [0.9608, 0.9564, 0.9788, 0.951, 0.8543, 0.9577, 0.9894] 
+2024-09-07 17:13:20.332093: Epoch time: 303.6 s 
+2024-09-07 17:13:23.444603:  
+2024-09-07 17:13:23.452359: Epoch 335 
+2024-09-07 17:13:23.453120: Current learning rate: 0.00693 
+2024-09-07 17:18:29.195244: train_loss -0.7447 
+2024-09-07 17:18:29.197700: val_loss -0.737 
+2024-09-07 17:18:29.198383: Pseudo dice [0.9587, 0.9542, 0.9733, 0.9444, 0.8704, 0.9629, 0.9905] 
+2024-09-07 17:18:29.198867: Epoch time: 305.75 s 
+2024-09-07 17:18:31.143310:  
+2024-09-07 17:18:31.144173: Epoch 336 
+2024-09-07 17:18:31.145186: Current learning rate: 0.00692 
+2024-09-07 17:23:26.146067: train_loss -0.7197 
+2024-09-07 17:23:26.148017: val_loss -0.7349 
+2024-09-07 17:23:26.148636: Pseudo dice [0.9557, 0.9505, 0.9709, 0.934, 0.856, 0.9637, 0.9873] 
+2024-09-07 17:23:26.150118: Epoch time: 295.01 s 
+2024-09-07 17:23:29.088973:  
+2024-09-07 17:23:29.104570: Epoch 337 
+2024-09-07 17:23:29.105495: Current learning rate: 0.00691 
+2024-09-07 17:28:14.209732: train_loss -0.7049 
+2024-09-07 17:28:14.216383: val_loss -0.6777 
+2024-09-07 17:28:14.218108: Pseudo dice [0.9549, 0.9516, 0.9609, 0.9386, 0.8509, 0.9612, 0.9892] 
+2024-09-07 17:28:14.219580: Epoch time: 285.13 s 
+2024-09-07 17:28:16.710897:  
+2024-09-07 17:28:16.712247: Epoch 338 
+2024-09-07 17:28:16.716460: Current learning rate: 0.0069 
+2024-09-07 17:33:00.319731: train_loss -0.7025 
+2024-09-07 17:33:00.321445: val_loss -0.7386 
+2024-09-07 17:33:00.322223: Pseudo dice [0.9577, 0.9531, 0.9728, 0.9382, 0.8395, 0.9537, 0.986] 
+2024-09-07 17:33:00.322852: Epoch time: 283.61 s 
+2024-09-07 17:33:02.646727:  
+2024-09-07 17:33:02.647617: Epoch 339 
+2024-09-07 17:33:02.648157: Current learning rate: 0.00689 
+2024-09-07 17:37:38.335892: train_loss -0.7324 
+2024-09-07 17:37:38.338752: val_loss -0.7298 
+2024-09-07 17:37:38.339359: Pseudo dice [0.9609, 0.9553, 0.9752, 0.9478, 0.8569, 0.9684, 0.9888] 
+2024-09-07 17:37:38.339828: Epoch time: 275.7 s 
+2024-09-07 17:37:40.745775:  
+2024-09-07 17:37:40.748784: Epoch 340 
+2024-09-07 17:37:40.749523: Current learning rate: 0.00688 
+2024-09-07 17:42:43.349739: train_loss -0.7186 
+2024-09-07 17:42:43.351878: val_loss -0.7419 
+2024-09-07 17:42:43.352903: Pseudo dice [0.9576, 0.9535, 0.9725, 0.9427, 0.8667, 0.9662, 0.9913] 
+2024-09-07 17:42:43.354307: Epoch time: 302.61 s 
+2024-09-07 17:42:45.683198:  
+2024-09-07 17:42:45.684571: Epoch 341 
+2024-09-07 17:42:45.686590: Current learning rate: 0.00687 
+2024-09-07 17:47:50.813298: train_loss -0.7146 
+2024-09-07 17:47:50.815315: val_loss -0.7488 
+2024-09-07 17:47:50.816007: Pseudo dice [0.9615, 0.9556, 0.9761, 0.9425, 0.857, 0.9658, 0.9879] 
+2024-09-07 17:47:50.816512: Epoch time: 305.14 s 
+2024-09-07 17:47:53.849271:  
+2024-09-07 17:47:53.850272: Epoch 342 
+2024-09-07 17:47:53.850903: Current learning rate: 0.00686 
+2024-09-07 17:52:38.542030: train_loss -0.7323 
+2024-09-07 17:52:38.544999: val_loss -0.7418 
+2024-09-07 17:52:38.545694: Pseudo dice [0.9616, 0.9543, 0.9764, 0.9402, 0.87, 0.969, 0.9921] 
+2024-09-07 17:52:38.546269: Epoch time: 284.7 s 
+2024-09-07 17:52:40.576671:  
+2024-09-07 17:52:40.578038: Epoch 343 
+2024-09-07 17:52:40.578753: Current learning rate: 0.00685 
+2024-09-07 17:57:28.543870: train_loss -0.7246 
+2024-09-07 17:57:28.546666: val_loss -0.726 
+2024-09-07 17:57:28.547240: Pseudo dice [0.9634, 0.9562, 0.977, 0.941, 0.8583, 0.9666, 0.9866] 
+2024-09-07 17:57:28.547739: Epoch time: 287.97 s 
+2024-09-07 17:57:30.929904:  
+2024-09-07 17:57:30.932262: Epoch 344 
+2024-09-07 17:57:30.932799: Current learning rate: 0.00684 
+2024-09-07 18:02:27.066391: train_loss -0.7369 
+2024-09-07 18:02:27.069017: val_loss -0.7629 
+2024-09-07 18:02:27.069769: Pseudo dice [0.9614, 0.954, 0.9762, 0.9418, 0.8554, 0.9619, 0.9892] 
+2024-09-07 18:02:27.070428: Epoch time: 296.14 s 
+2024-09-07 18:02:29.028707:  
+2024-09-07 18:02:29.029940: Epoch 345 
+2024-09-07 18:02:29.030596: Current learning rate: 0.00683 
+2024-09-07 18:07:26.449173: train_loss -0.7175 
+2024-09-07 18:07:26.451295: val_loss -0.7544 
+2024-09-07 18:07:26.452050: Pseudo dice [0.9613, 0.9536, 0.9744, 0.9414, 0.8735, 0.9677, 0.99] 
+2024-09-07 18:07:26.452772: Epoch time: 297.43 s 
+2024-09-07 18:07:28.887923:  
+2024-09-07 18:07:28.888702: Epoch 346 
+2024-09-07 18:07:28.889186: Current learning rate: 0.00682 
+2024-09-07 18:12:29.947708: train_loss -0.7224 
+2024-09-07 18:12:29.949685: val_loss -0.7766 
+2024-09-07 18:12:29.951088: Pseudo dice [0.9637, 0.9557, 0.9764, 0.9458, 0.8738, 0.9697, 0.9901] 
+2024-09-07 18:12:29.953254: Epoch time: 301.06 s 
+2024-09-07 18:12:32.358244:  
+2024-09-07 18:12:32.361089: Epoch 347 
+2024-09-07 18:12:32.361938: Current learning rate: 0.00681 
+2024-09-07 18:17:19.423126: train_loss -0.742 
+2024-09-07 18:17:19.425624: val_loss -0.7493 
+2024-09-07 18:17:19.426346: Pseudo dice [0.9639, 0.9549, 0.9793, 0.9494, 0.8767, 0.9683, 0.9924] 
+2024-09-07 18:17:19.427009: Epoch time: 287.07 s 
+2024-09-07 18:17:21.716196:  
+2024-09-07 18:17:21.718820: Epoch 348 
+2024-09-07 18:17:21.724336: Current learning rate: 0.0068 
+2024-09-07 18:22:06.737370: train_loss -0.7314 
+2024-09-07 18:22:06.739894: val_loss -0.7368 
+2024-09-07 18:22:06.740448: Pseudo dice [0.9635, 0.9552, 0.9789, 0.9502, 0.8843, 0.9687, 0.991] 
+2024-09-07 18:22:06.741175: Epoch time: 285.03 s 
+2024-09-07 18:22:06.741831: Yayy! New best EMA pseudo Dice: 0.9509 
+2024-09-07 18:22:13.061412:  
+2024-09-07 18:22:13.062170: Epoch 349 
+2024-09-07 18:22:13.062658: Current learning rate: 0.0068 
+2024-09-07 18:26:58.115860: train_loss -0.7425 
+2024-09-07 18:26:58.117644: val_loss -0.7282 
+2024-09-07 18:26:58.118496: Pseudo dice [0.9615, 0.9565, 0.9747, 0.9453, 0.8679, 0.9665, 0.9911] 
+2024-09-07 18:26:58.119119: Epoch time: 285.06 s 
+2024-09-07 18:26:59.997838: Yayy! New best EMA pseudo Dice: 0.951 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_7_22_04_47.txt

@@ -0,0 +1,379 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-07 22:04:50.436455: Using torch.compile... 
+2024-09-07 22:04:57.912631: do_dummy_2d_data_aug: False 
+2024-09-07 22:04:57.924833: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-07 22:04:57.925860: The split file contains 5 splits. 
+2024-09-07 22:04:57.926275: Desired fold for training: 0 
+2024-09-07 22:04:57.926643: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-07 22:05:14.131174: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-07 22:05:15.006320:  
+2024-09-07 22:05:15.023768: Epoch 350 
+2024-09-07 22:05:15.025942: Current learning rate: 0.00679 
+2024-09-07 22:20:03.486652: train_loss -0.7348 
+2024-09-07 22:20:03.489151: val_loss -0.7481 
+2024-09-07 22:20:03.489682: Pseudo dice [0.9619, 0.9576, 0.976, 0.9498, 0.8646, 0.9674, 0.9882] 
+2024-09-07 22:20:03.490121: Epoch time: 888.49 s 
+2024-09-07 22:20:03.490541: Yayy! New best EMA pseudo Dice: 0.9512 
+2024-09-07 22:20:07.474930:  
+2024-09-07 22:20:07.475547: Epoch 351 
+2024-09-07 22:20:07.476246: Current learning rate: 0.00678 
+2024-09-07 22:24:52.430929: train_loss -0.7264 
+2024-09-07 22:24:52.447058: val_loss -0.7304 
+2024-09-07 22:24:52.447796: Pseudo dice [0.9615, 0.9567, 0.9744, 0.9448, 0.8601, 0.9675, 0.9886] 
+2024-09-07 22:24:52.448460: Epoch time: 284.96 s 
+2024-09-07 22:24:56.069815:  
+2024-09-07 22:24:56.070733: Epoch 352 
+2024-09-07 22:24:56.071400: Current learning rate: 0.00677 
+2024-09-07 22:29:27.276925: train_loss -0.7275 
+2024-09-07 22:29:27.279526: val_loss -0.7501 
+2024-09-07 22:29:27.280204: Pseudo dice [0.9652, 0.9598, 0.979, 0.9458, 0.8709, 0.9619, 0.9891] 
+2024-09-07 22:29:27.280817: Epoch time: 271.21 s 
+2024-09-07 22:29:27.281518: Yayy! New best EMA pseudo Dice: 0.9513 
+2024-09-07 22:29:35.233615:  
+2024-09-07 22:29:35.240188: Epoch 353 
+2024-09-07 22:29:35.240692: Current learning rate: 0.00676 
+2024-09-07 22:34:01.228054: train_loss -0.7313 
+2024-09-07 22:34:01.240554: val_loss -0.743 
+2024-09-07 22:34:01.241293: Pseudo dice [0.9657, 0.9575, 0.9798, 0.9481, 0.8789, 0.9708, 0.9919] 
+2024-09-07 22:34:01.241909: Epoch time: 266.0 s 
+2024-09-07 22:34:01.242370: Yayy! New best EMA pseudo Dice: 0.9518 
+2024-09-07 22:34:07.361151:  
+2024-09-07 22:34:07.362123: Epoch 354 
+2024-09-07 22:34:07.362683: Current learning rate: 0.00675 
+2024-09-07 22:38:32.082459: train_loss -0.7235 
+2024-09-07 22:38:32.107754: val_loss -0.7235 
+2024-09-07 22:38:32.108586: Pseudo dice [0.9649, 0.9579, 0.9799, 0.9517, 0.8419, 0.9627, 0.9867] 
+2024-09-07 22:38:32.109203: Epoch time: 264.73 s 
+2024-09-07 22:38:36.039857:  
+2024-09-07 22:38:36.052402: Epoch 355 
+2024-09-07 22:38:36.064449: Current learning rate: 0.00674 
+2024-09-07 22:43:24.784304: train_loss -0.7339 
+2024-09-07 22:43:24.803568: val_loss -0.7074 
+2024-09-07 22:43:24.804627: Pseudo dice [0.9579, 0.9523, 0.9724, 0.9408, 0.8685, 0.9661, 0.9878] 
+2024-09-07 22:43:24.806409: Epoch time: 288.75 s 
+2024-09-07 22:43:27.290543:  
+2024-09-07 22:43:27.304433: Epoch 356 
+2024-09-07 22:43:27.306978: Current learning rate: 0.00673 
+2024-09-07 22:48:19.434531: train_loss -0.7334 
+2024-09-07 22:48:19.449667: val_loss -0.7314 
+2024-09-07 22:48:19.450305: Pseudo dice [0.9626, 0.9577, 0.9761, 0.9516, 0.8708, 0.9616, 0.9906] 
+2024-09-07 22:48:19.450804: Epoch time: 292.15 s 
+2024-09-07 22:48:22.621067:  
+2024-09-07 22:48:22.622461: Epoch 357 
+2024-09-07 22:48:22.623893: Current learning rate: 0.00672 
+2024-09-07 22:53:26.883884: train_loss -0.7379 
+2024-09-07 22:53:26.886493: val_loss -0.7662 
+2024-09-07 22:53:26.887113: Pseudo dice [0.9647, 0.959, 0.9808, 0.9482, 0.8637, 0.9649, 0.9892] 
+2024-09-07 22:53:26.887936: Epoch time: 304.27 s 
+2024-09-07 22:53:28.638150:  
+2024-09-07 22:53:28.639353: Epoch 358 
+2024-09-07 22:53:28.641742: Current learning rate: 0.00671 
+2024-09-07 22:58:22.097706: train_loss -0.7445 
+2024-09-07 22:58:22.104040: val_loss -0.7524 
+2024-09-07 22:58:22.104566: Pseudo dice [0.9631, 0.9584, 0.9776, 0.9386, 0.8725, 0.9673, 0.99] 
+2024-09-07 22:58:22.106256: Epoch time: 293.46 s 
+2024-09-07 22:58:23.731670:  
+2024-09-07 22:58:23.732604: Epoch 359 
+2024-09-07 22:58:23.734023: Current learning rate: 0.0067 
+2024-09-07 23:02:55.951777: train_loss -0.7466 
+2024-09-07 23:02:55.953769: val_loss -0.7159 
+2024-09-07 23:02:55.954276: Pseudo dice [0.9631, 0.9528, 0.9781, 0.9445, 0.8693, 0.9669, 0.992] 
+2024-09-07 23:02:55.954754: Epoch time: 272.22 s 
+2024-09-07 23:02:55.955176: Yayy! New best EMA pseudo Dice: 0.9518 
+2024-09-07 23:02:59.629727:  
+2024-09-07 23:02:59.630902: Epoch 360 
+2024-09-07 23:02:59.632901: Current learning rate: 0.00669 
+2024-09-07 23:07:44.682072: train_loss -0.7207 
+2024-09-07 23:07:44.683780: val_loss -0.734 
+2024-09-07 23:07:44.685427: Pseudo dice [0.9662, 0.9599, 0.981, 0.9546, 0.8727, 0.9664, 0.9872] 
+2024-09-07 23:07:44.686458: Epoch time: 285.06 s 
+2024-09-07 23:07:44.688083: Yayy! New best EMA pseudo Dice: 0.9522 
+2024-09-07 23:07:48.196903:  
+2024-09-07 23:07:48.202654: Epoch 361 
+2024-09-07 23:07:48.203948: Current learning rate: 0.00668 
+2024-09-07 23:12:17.617336: train_loss -0.7371 
+2024-09-07 23:12:17.633031: val_loss -0.7174 
+2024-09-07 23:12:17.634238: Pseudo dice [0.9632, 0.9564, 0.9793, 0.9444, 0.8828, 0.9685, 0.9914] 
+2024-09-07 23:12:17.635134: Epoch time: 269.43 s 
+2024-09-07 23:12:17.636246: Yayy! New best EMA pseudo Dice: 0.9525 
+2024-09-07 23:12:21.628394:  
+2024-09-07 23:12:21.629321: Epoch 362 
+2024-09-07 23:12:21.630131: Current learning rate: 0.00667 
+2024-09-07 23:16:49.394240: train_loss -0.752 
+2024-09-07 23:16:49.396390: val_loss -0.7295 
+2024-09-07 23:16:49.396987: Pseudo dice [0.9599, 0.9577, 0.9756, 0.9497, 0.8582, 0.9653, 0.9873] 
+2024-09-07 23:16:49.397500: Epoch time: 267.77 s 
+2024-09-07 23:16:51.538909:  
+2024-09-07 23:16:51.539838: Epoch 363 
+2024-09-07 23:16:51.540418: Current learning rate: 0.00666 
+2024-09-07 23:21:29.090170: train_loss -0.7278 
+2024-09-07 23:21:29.092506: val_loss -0.7248 
+2024-09-07 23:21:29.093424: Pseudo dice [0.9621, 0.9545, 0.9756, 0.9475, 0.865, 0.9647, 0.9907] 
+2024-09-07 23:21:29.094384: Epoch time: 277.56 s 
+2024-09-07 23:21:31.086623:  
+2024-09-07 23:21:31.087512: Epoch 364 
+2024-09-07 23:21:31.088445: Current learning rate: 0.00665 
+2024-09-07 23:26:15.183840: train_loss -0.7379 
+2024-09-07 23:26:15.186132: val_loss -0.7314 
+2024-09-07 23:26:15.186718: Pseudo dice [0.9623, 0.9551, 0.9768, 0.9476, 0.8739, 0.9688, 0.9914] 
+2024-09-07 23:26:15.187627: Epoch time: 284.1 s 
+2024-09-07 23:26:20.044234:  
+2024-09-07 23:26:20.046443: Epoch 365 
+2024-09-07 23:26:20.047663: Current learning rate: 0.00665 
+2024-09-07 23:31:06.510001: train_loss -0.7204 
+2024-09-07 23:31:06.512149: val_loss -0.7488 
+2024-09-07 23:31:06.512859: Pseudo dice [0.9583, 0.9522, 0.9684, 0.9422, 0.8601, 0.9574, 0.9893] 
+2024-09-07 23:31:06.513555: Epoch time: 286.47 s 
+2024-09-07 23:31:08.417214:  
+2024-09-07 23:31:08.417883: Epoch 366 
+2024-09-07 23:31:08.418654: Current learning rate: 0.00664 
+2024-09-07 23:36:03.652073: train_loss -0.7356 
+2024-09-07 23:36:03.658689: val_loss -0.7349 
+2024-09-07 23:36:03.663128: Pseudo dice [0.9601, 0.9572, 0.9769, 0.9443, 0.8748, 0.9678, 0.988] 
+2024-09-07 23:36:03.665215: Epoch time: 295.24 s 
+2024-09-07 23:36:05.906566:  
+2024-09-07 23:36:05.908732: Epoch 367 
+2024-09-07 23:36:05.910182: Current learning rate: 0.00663 
+2024-09-07 23:40:55.081939: train_loss -0.7548 
+2024-09-07 23:40:55.083933: val_loss -0.7278 
+2024-09-07 23:40:55.084474: Pseudo dice [0.9629, 0.9615, 0.9754, 0.9506, 0.8683, 0.9654, 0.9893] 
+2024-09-07 23:40:55.085031: Epoch time: 289.18 s 
+2024-09-07 23:40:57.238287:  
+2024-09-07 23:40:57.239110: Epoch 368 
+2024-09-07 23:40:57.239876: Current learning rate: 0.00662 
+2024-09-07 23:46:09.441271: train_loss -0.7418 
+2024-09-07 23:46:09.454659: val_loss -0.7254 
+2024-09-07 23:46:09.456121: Pseudo dice [0.9631, 0.9559, 0.979, 0.9471, 0.8706, 0.9652, 0.9906] 
+2024-09-07 23:46:09.457182: Epoch time: 312.21 s 
+2024-09-07 23:46:11.970695:  
+2024-09-07 23:46:11.971725: Epoch 369 
+2024-09-07 23:46:11.972404: Current learning rate: 0.00661 
+2024-09-07 23:52:16.300404: train_loss -0.74 
+2024-09-07 23:52:16.303117: val_loss -0.7248 
+2024-09-07 23:52:16.304048: Pseudo dice [0.9601, 0.9541, 0.9756, 0.9463, 0.8715, 0.9664, 0.9911] 
+2024-09-07 23:52:16.304669: Epoch time: 364.34 s 
+2024-09-07 23:52:18.643195:  
+2024-09-07 23:52:18.643970: Epoch 370 
+2024-09-07 23:52:18.644596: Current learning rate: 0.0066 
+2024-09-07 23:58:01.773705: train_loss -0.7218 
+2024-09-07 23:58:01.776166: val_loss -0.7479 
+2024-09-07 23:58:01.777072: Pseudo dice [0.9636, 0.9577, 0.9788, 0.948, 0.8776, 0.9719, 0.9932] 
+2024-09-07 23:58:01.777585: Epoch time: 343.14 s 
+2024-09-07 23:58:01.778045: Yayy! New best EMA pseudo Dice: 0.9525 
+2024-09-07 23:58:07.947380:  
+2024-09-07 23:58:07.948505: Epoch 371 
+2024-09-07 23:58:07.949520: Current learning rate: 0.00659 
+2024-09-08 00:03:44.083137: train_loss -0.7363 
+2024-09-08 00:03:44.085596: val_loss -0.7215 
+2024-09-08 00:03:44.086218: Pseudo dice [0.9636, 0.9559, 0.9788, 0.946, 0.8614, 0.9585, 0.9884] 
+2024-09-08 00:03:44.086729: Epoch time: 336.14 s 
+2024-09-08 00:03:46.818362:  
+2024-09-08 00:03:46.819499: Epoch 372 
+2024-09-08 00:03:46.820522: Current learning rate: 0.00658 
+2024-09-08 00:09:32.929209: train_loss -0.7332 
+2024-09-08 00:09:32.932279: val_loss -0.745 
+2024-09-08 00:09:32.932979: Pseudo dice [0.9642, 0.9584, 0.9767, 0.952, 0.8837, 0.9693, 0.992] 
+2024-09-08 00:09:32.933601: Epoch time: 346.12 s 
+2024-09-08 00:09:32.934094: Yayy! New best EMA pseudo Dice: 0.9527 
+2024-09-08 00:09:37.829674:  
+2024-09-08 00:09:37.830686: Epoch 373 
+2024-09-08 00:09:37.831239: Current learning rate: 0.00657 
+2024-09-08 00:14:49.411239: train_loss -0.7291 
+2024-09-08 00:14:49.413806: val_loss -0.7112 
+2024-09-08 00:14:49.414559: Pseudo dice [0.9575, 0.9553, 0.978, 0.9379, 0.8685, 0.9532, 0.99] 
+2024-09-08 00:14:49.415106: Epoch time: 311.59 s 
+2024-09-08 00:14:51.810179:  
+2024-09-08 00:14:51.812260: Epoch 374 
+2024-09-08 00:14:51.813382: Current learning rate: 0.00656 
+2024-09-08 00:19:44.990911: train_loss -0.7211 
+2024-09-08 00:19:45.006667: val_loss -0.7234 
+2024-09-08 00:19:45.007273: Pseudo dice [0.9609, 0.9545, 0.9755, 0.9469, 0.8735, 0.9656, 0.9912] 
+2024-09-08 00:19:45.007875: Epoch time: 293.2 s 
+2024-09-08 00:19:47.548718:  
+2024-09-08 00:19:47.549706: Epoch 375 
+2024-09-08 00:19:47.551049: Current learning rate: 0.00655 
+2024-09-08 00:24:52.217963: train_loss -0.7212 
+2024-09-08 00:24:52.220148: val_loss -0.7087 
+2024-09-08 00:24:52.220783: Pseudo dice [0.961, 0.9549, 0.9672, 0.9458, 0.8527, 0.9654, 0.9896] 
+2024-09-08 00:24:52.221347: Epoch time: 304.68 s 
+2024-09-08 00:24:54.139486:  
+2024-09-08 00:24:54.140689: Epoch 376 
+2024-09-08 00:24:54.141441: Current learning rate: 0.00654 
+2024-09-08 00:29:34.073451: train_loss -0.7236 
+2024-09-08 00:29:34.077057: val_loss -0.7045 
+2024-09-08 00:29:34.077683: Pseudo dice [0.9596, 0.9566, 0.9734, 0.9353, 0.8675, 0.9682, 0.9903] 
+2024-09-08 00:29:34.078281: Epoch time: 279.94 s 
+2024-09-08 00:29:36.399098:  
+2024-09-08 00:29:36.400293: Epoch 377 
+2024-09-08 00:29:36.401027: Current learning rate: 0.00653 
+2024-09-08 00:34:31.736445: train_loss -0.727 
+2024-09-08 00:34:31.738098: val_loss -0.7567 
+2024-09-08 00:34:31.739261: Pseudo dice [0.9622, 0.9562, 0.9781, 0.9447, 0.8681, 0.9651, 0.9897] 
+2024-09-08 00:34:31.740235: Epoch time: 295.34 s 
+2024-09-08 00:34:35.439516:  
+2024-09-08 00:34:35.443596: Epoch 378 
+2024-09-08 00:34:35.444572: Current learning rate: 0.00652 
+2024-09-08 00:39:31.576055: train_loss -0.7202 
+2024-09-08 00:39:31.577361: val_loss -0.7386 
+2024-09-08 00:39:31.578000: Pseudo dice [0.964, 0.9571, 0.9798, 0.9435, 0.8666, 0.9643, 0.9902] 
+2024-09-08 00:39:31.578533: Epoch time: 296.14 s 
+2024-09-08 00:39:33.191946:  
+2024-09-08 00:39:33.193068: Epoch 379 
+2024-09-08 00:39:33.193743: Current learning rate: 0.00651 
+2024-09-08 00:44:09.351942: train_loss -0.7283 
+2024-09-08 00:44:09.353483: val_loss -0.7226 
+2024-09-08 00:44:09.354486: Pseudo dice [0.9634, 0.9533, 0.98, 0.9418, 0.8436, 0.9641, 0.9889] 
+2024-09-08 00:44:09.355370: Epoch time: 276.16 s 
+2024-09-08 00:44:11.640047:  
+2024-09-08 00:44:11.641275: Epoch 380 
+2024-09-08 00:44:11.642010: Current learning rate: 0.0065 
+2024-09-08 00:49:13.149114: train_loss -0.7346 
+2024-09-08 00:49:13.152743: val_loss -0.7316 
+2024-09-08 00:49:13.153794: Pseudo dice [0.96, 0.9541, 0.9719, 0.9432, 0.8672, 0.966, 0.9899] 
+2024-09-08 00:49:13.156345: Epoch time: 301.52 s 
+2024-09-08 00:49:15.439008:  
+2024-09-08 00:49:15.439958: Epoch 381 
+2024-09-08 00:49:15.440609: Current learning rate: 0.00649 
+2024-09-08 00:54:00.546713: train_loss -0.7272 
+2024-09-08 00:54:00.547955: val_loss -0.7798 
+2024-09-08 00:54:00.548475: Pseudo dice [0.9616, 0.9561, 0.9783, 0.9464, 0.8654, 0.9661, 0.9895] 
+2024-09-08 00:54:00.549080: Epoch time: 285.11 s 
+2024-09-08 00:54:02.546525:  
+2024-09-08 00:54:02.547561: Epoch 382 
+2024-09-08 00:54:02.548214: Current learning rate: 0.00648 
+2024-09-08 00:58:58.576061: train_loss -0.719 
+2024-09-08 00:58:58.578232: val_loss -0.7448 
+2024-09-08 00:58:58.578677: Pseudo dice [0.9617, 0.9563, 0.9791, 0.9484, 0.8731, 0.9678, 0.9912] 
+2024-09-08 00:58:58.579104: Epoch time: 296.04 s 
+2024-09-08 00:59:00.221547:  
+2024-09-08 00:59:00.222251: Epoch 383 
+2024-09-08 00:59:00.223145: Current learning rate: 0.00648 
+2024-09-08 01:04:03.407910: train_loss -0.7293 
+2024-09-08 01:04:03.410652: val_loss -0.7395 
+2024-09-08 01:04:03.411332: Pseudo dice [0.9609, 0.9561, 0.9775, 0.9481, 0.8727, 0.9671, 0.9909] 
+2024-09-08 01:04:03.411814: Epoch time: 303.19 s 
+2024-09-08 01:04:05.816084:  
+2024-09-08 01:04:05.817439: Epoch 384 
+2024-09-08 01:04:05.818743: Current learning rate: 0.00647 
+2024-09-08 01:09:06.522553: train_loss -0.7213 
+2024-09-08 01:09:06.525675: val_loss -0.7314 
+2024-09-08 01:09:06.526363: Pseudo dice [0.9623, 0.9583, 0.9767, 0.9484, 0.8666, 0.9699, 0.9905] 
+2024-09-08 01:09:06.527015: Epoch time: 300.71 s 
+2024-09-08 01:09:08.057703:  
+2024-09-08 01:09:08.058463: Epoch 385 
+2024-09-08 01:09:08.058985: Current learning rate: 0.00646 
+2024-09-08 01:13:49.323172: train_loss -0.7264 
+2024-09-08 01:13:49.337723: val_loss -0.7348 
+2024-09-08 01:13:49.338467: Pseudo dice [0.9607, 0.9528, 0.9767, 0.944, 0.8583, 0.9673, 0.9879] 
+2024-09-08 01:13:49.339068: Epoch time: 281.27 s 
+2024-09-08 01:13:51.685326:  
+2024-09-08 01:13:51.686450: Epoch 386 
+2024-09-08 01:13:51.687132: Current learning rate: 0.00645 
+2024-09-08 01:18:40.290848: train_loss -0.7381 
+2024-09-08 01:18:40.297343: val_loss -0.7599 
+2024-09-08 01:18:40.298711: Pseudo dice [0.9613, 0.9562, 0.9776, 0.9493, 0.8708, 0.97, 0.9901] 
+2024-09-08 01:18:40.300279: Epoch time: 288.61 s 
+2024-09-08 01:18:42.402846:  
+2024-09-08 01:18:42.403584: Epoch 387 
+2024-09-08 01:18:42.404164: Current learning rate: 0.00644 
+2024-09-08 01:23:24.243913: train_loss -0.7385 
+2024-09-08 01:23:24.246147: val_loss -0.7283 
+2024-09-08 01:23:24.246714: Pseudo dice [0.9626, 0.96, 0.9767, 0.9487, 0.872, 0.9663, 0.9917] 
+2024-09-08 01:23:24.247227: Epoch time: 281.84 s 
+2024-09-08 01:23:26.284719:  
+2024-09-08 01:23:26.285917: Epoch 388 
+2024-09-08 01:23:26.286703: Current learning rate: 0.00643 
+2024-09-08 01:27:52.871003: train_loss -0.7216 
+2024-09-08 01:27:52.874235: val_loss -0.699 
+2024-09-08 01:27:52.875526: Pseudo dice [0.9616, 0.9568, 0.9751, 0.9458, 0.8651, 0.9666, 0.9895] 
+2024-09-08 01:27:52.876401: Epoch time: 266.59 s 
+2024-09-08 01:27:56.437101:  
+2024-09-08 01:27:56.438122: Epoch 389 
+2024-09-08 01:27:56.438797: Current learning rate: 0.00642 
+2024-09-08 01:32:40.023240: train_loss -0.7338 
+2024-09-08 01:32:40.025555: val_loss -0.7345 
+2024-09-08 01:32:40.026439: Pseudo dice [0.9629, 0.9555, 0.9802, 0.9434, 0.8781, 0.9665, 0.9913] 
+2024-09-08 01:32:40.027032: Epoch time: 283.59 s 
+2024-09-08 01:32:42.271252:  
+2024-09-08 01:32:42.272675: Epoch 390 
+2024-09-08 01:32:42.273480: Current learning rate: 0.00641 
+2024-09-08 01:37:09.879209: train_loss -0.7424 
+2024-09-08 01:37:09.881777: val_loss -0.7194 
+2024-09-08 01:37:09.882622: Pseudo dice [0.9638, 0.9586, 0.9756, 0.9451, 0.863, 0.9655, 0.9897] 
+2024-09-08 01:37:09.883272: Epoch time: 267.61 s 
+2024-09-08 01:37:14.029768:  
+2024-09-08 01:37:14.030790: Epoch 391 
+2024-09-08 01:37:14.031358: Current learning rate: 0.0064 
+2024-09-08 01:41:56.367385: train_loss -0.7513 
+2024-09-08 01:41:56.369679: val_loss -0.7239 
+2024-09-08 01:41:56.370354: Pseudo dice [0.9617, 0.9536, 0.9795, 0.9457, 0.8671, 0.9686, 0.9916] 
+2024-09-08 01:41:56.370860: Epoch time: 282.34 s 
+2024-09-08 01:41:58.451717:  
+2024-09-08 01:41:58.452411: Epoch 392 
+2024-09-08 01:41:58.452900: Current learning rate: 0.00639 
+2024-09-08 01:46:50.131638: train_loss -0.7099 
+2024-09-08 01:46:50.134486: val_loss -0.718 
+2024-09-08 01:46:50.135189: Pseudo dice [0.9585, 0.9512, 0.9781, 0.9316, 0.8623, 0.9625, 0.9893] 
+2024-09-08 01:46:50.135905: Epoch time: 291.68 s 
+2024-09-08 01:46:52.147480:  
+2024-09-08 01:46:52.148285: Epoch 393 
+2024-09-08 01:46:52.148881: Current learning rate: 0.00638 
+2024-09-08 01:51:44.209609: train_loss -0.7224 
+2024-09-08 01:51:44.221107: val_loss -0.7612 
+2024-09-08 01:51:44.222059: Pseudo dice [0.9631, 0.9578, 0.9786, 0.9489, 0.8631, 0.967, 0.9897] 
+2024-09-08 01:51:44.222746: Epoch time: 292.07 s 
+2024-09-08 01:51:47.007719:  
+2024-09-08 01:51:47.008863: Epoch 394 
+2024-09-08 01:51:47.009501: Current learning rate: 0.00637 
+2024-09-08 01:56:42.218983: train_loss -0.7286 
+2024-09-08 01:56:42.221816: val_loss -0.7328 
+2024-09-08 01:56:42.222394: Pseudo dice [0.9637, 0.9594, 0.9785, 0.9509, 0.8781, 0.968, 0.9903] 
+2024-09-08 01:56:42.222859: Epoch time: 295.22 s 
+2024-09-08 01:56:44.163985:  
+2024-09-08 01:56:44.165304: Epoch 395 
+2024-09-08 01:56:44.166382: Current learning rate: 0.00636 
+2024-09-08 02:01:33.682235: train_loss -0.7097 
+2024-09-08 02:01:33.684346: val_loss -0.7484 
+2024-09-08 02:01:33.685336: Pseudo dice [0.957, 0.9533, 0.9741, 0.9382, 0.8616, 0.9626, 0.9898] 
+2024-09-08 02:01:33.686122: Epoch time: 289.52 s 
+2024-09-08 02:01:36.712077:  
+2024-09-08 02:01:36.713010: Epoch 396 
+2024-09-08 02:01:36.713566: Current learning rate: 0.00635 
+2024-09-08 02:06:13.505131: train_loss -0.7329 
+2024-09-08 02:06:13.507925: val_loss -0.7732 
+2024-09-08 02:06:13.508677: Pseudo dice [0.9628, 0.9541, 0.979, 0.9408, 0.8714, 0.9708, 0.9908] 
+2024-09-08 02:06:13.509211: Epoch time: 276.8 s 
+2024-09-08 02:06:15.560633:  
+2024-09-08 02:06:15.561455: Epoch 397 
+2024-09-08 02:06:15.562352: Current learning rate: 0.00634 
+2024-09-08 02:11:07.614176: train_loss -0.7104 
+2024-09-08 02:11:07.616350: val_loss -0.7455 
+2024-09-08 02:11:07.617117: Pseudo dice [0.9618, 0.9567, 0.9754, 0.9468, 0.8654, 0.9656, 0.9903] 
+2024-09-08 02:11:07.617598: Epoch time: 292.06 s 
+2024-09-08 02:11:10.406317:  
+2024-09-08 02:11:10.407481: Epoch 398 
+2024-09-08 02:11:10.408231: Current learning rate: 0.00633 
+2024-09-08 02:16:30.152310: train_loss -0.7291 
+2024-09-08 02:16:30.157570: val_loss -0.7392 
+2024-09-08 02:16:30.158858: Pseudo dice [0.9614, 0.9553, 0.976, 0.942, 0.8669, 0.9647, 0.9916] 
+2024-09-08 02:16:30.159933: Epoch time: 319.75 s 
+2024-09-08 02:16:32.820824:  
+2024-09-08 02:16:32.822403: Epoch 399 
+2024-09-08 02:16:32.824252: Current learning rate: 0.00632 
+2024-09-08 02:21:28.454255: train_loss -0.7168 
+2024-09-08 02:21:28.457417: val_loss -0.7374 
+2024-09-08 02:21:28.458918: Pseudo dice [0.9624, 0.9553, 0.9753, 0.9445, 0.8561, 0.9632, 0.988] 
+2024-09-08 02:21:28.459815: Epoch time: 295.64 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_8_06_04_18.txt

@@ -0,0 +1,376 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-08 06:04:22.193567: Using torch.compile... 
+2024-09-08 06:04:28.562759: do_dummy_2d_data_aug: False 
+2024-09-08 06:04:28.577198: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-08 06:04:28.579905: The split file contains 5 splits. 
+2024-09-08 06:04:28.581280: Desired fold for training: 0 
+2024-09-08 06:04:28.582640: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-08 06:04:40.483038: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-08 06:04:41.596569:  
+2024-09-08 06:04:41.602085: Epoch 400 
+2024-09-08 06:04:41.604947: Current learning rate: 0.00631 
+2024-09-08 06:19:31.794949: train_loss -0.7402 
+2024-09-08 06:19:31.808858: val_loss -0.704 
+2024-09-08 06:19:31.809546: Pseudo dice [0.9621, 0.951, 0.9777, 0.9489, 0.8541, 0.9672, 0.9894] 
+2024-09-08 06:19:31.810093: Epoch time: 890.21 s 
+2024-09-08 06:19:34.378989:  
+2024-09-08 06:19:34.379738: Epoch 401 
+2024-09-08 06:19:34.380462: Current learning rate: 0.0063 
+2024-09-08 06:23:48.994299: train_loss -0.7224 
+2024-09-08 06:23:49.005950: val_loss -0.7473 
+2024-09-08 06:23:49.007166: Pseudo dice [0.9613, 0.9568, 0.9762, 0.9414, 0.8494, 0.9644, 0.989] 
+2024-09-08 06:23:49.008452: Epoch time: 254.62 s 
+2024-09-08 06:23:51.630479:  
+2024-09-08 06:23:51.632395: Epoch 402 
+2024-09-08 06:23:51.633239: Current learning rate: 0.0063 
+2024-09-08 06:28:26.933783: train_loss -0.7253 
+2024-09-08 06:28:26.948979: val_loss -0.7277 
+2024-09-08 06:28:26.952197: Pseudo dice [0.9639, 0.9589, 0.9757, 0.9493, 0.8742, 0.966, 0.9912] 
+2024-09-08 06:28:26.952993: Epoch time: 275.31 s 
+2024-09-08 06:28:31.224618:  
+2024-09-08 06:28:31.226063: Epoch 403 
+2024-09-08 06:28:31.227442: Current learning rate: 0.00629 
+2024-09-08 06:33:11.356989: train_loss -0.7381 
+2024-09-08 06:33:11.359057: val_loss -0.7065 
+2024-09-08 06:33:11.359636: Pseudo dice [0.9564, 0.9539, 0.968, 0.9471, 0.8674, 0.9641, 0.9884] 
+2024-09-08 06:33:11.360134: Epoch time: 280.14 s 
+2024-09-08 06:33:18.352405:  
+2024-09-08 06:33:18.353296: Epoch 404 
+2024-09-08 06:33:18.354028: Current learning rate: 0.00628 
+2024-09-08 06:38:10.204727: train_loss -0.7163 
+2024-09-08 06:38:10.220631: val_loss -0.7647 
+2024-09-08 06:38:10.221112: Pseudo dice [0.9636, 0.9571, 0.9778, 0.9434, 0.869, 0.9712, 0.991] 
+2024-09-08 06:38:10.221553: Epoch time: 291.86 s 
+2024-09-08 06:38:13.816221:  
+2024-09-08 06:38:13.831028: Epoch 405 
+2024-09-08 06:38:13.832027: Current learning rate: 0.00627 
+2024-09-08 06:43:01.659381: train_loss -0.7474 
+2024-09-08 06:43:01.672629: val_loss -0.7452 
+2024-09-08 06:43:01.673951: Pseudo dice [0.9617, 0.9573, 0.9762, 0.947, 0.8761, 0.9706, 0.9915] 
+2024-09-08 06:43:01.675949: Epoch time: 287.85 s 
+2024-09-08 06:43:05.849131:  
+2024-09-08 06:43:05.850045: Epoch 406 
+2024-09-08 06:43:05.851887: Current learning rate: 0.00626 
+2024-09-08 06:47:29.177991: train_loss -0.753 
+2024-09-08 06:47:29.195497: val_loss -0.7231 
+2024-09-08 06:47:29.196134: Pseudo dice [0.9585, 0.9567, 0.968, 0.936, 0.8723, 0.9598, 0.9911] 
+2024-09-08 06:47:29.196585: Epoch time: 263.33 s 
+2024-09-08 06:47:33.212547:  
+2024-09-08 06:47:33.222683: Epoch 407 
+2024-09-08 06:47:33.224624: Current learning rate: 0.00625 
+2024-09-08 06:52:01.350911: train_loss -0.7395 
+2024-09-08 06:52:01.361780: val_loss -0.725 
+2024-09-08 06:52:01.362455: Pseudo dice [0.9594, 0.9541, 0.9766, 0.9466, 0.8586, 0.9639, 0.9896] 
+2024-09-08 06:52:01.362957: Epoch time: 268.15 s 
+2024-09-08 06:52:03.939089:  
+2024-09-08 06:52:03.940401: Epoch 408 
+2024-09-08 06:52:03.941464: Current learning rate: 0.00624 
+2024-09-08 06:56:51.567301: train_loss -0.7249 
+2024-09-08 06:56:51.571867: val_loss -0.7539 
+2024-09-08 06:56:51.573717: Pseudo dice [0.9631, 0.9546, 0.9798, 0.9502, 0.8742, 0.9685, 0.9898] 
+2024-09-08 06:56:51.576737: Epoch time: 287.63 s 
+2024-09-08 06:56:54.278836:  
+2024-09-08 06:56:54.290875: Epoch 409 
+2024-09-08 06:56:54.294496: Current learning rate: 0.00623 
+2024-09-08 07:01:32.565291: train_loss -0.7403 
+2024-09-08 07:01:32.604045: val_loss -0.7368 
+2024-09-08 07:01:32.604958: Pseudo dice [0.9608, 0.9581, 0.9751, 0.9468, 0.8797, 0.9673, 0.989] 
+2024-09-08 07:01:32.605963: Epoch time: 278.29 s 
+2024-09-08 07:01:36.423521:  
+2024-09-08 07:01:36.431705: Epoch 410 
+2024-09-08 07:01:36.432385: Current learning rate: 0.00622 
+2024-09-08 07:06:22.141404: train_loss -0.7393 
+2024-09-08 07:06:22.167896: val_loss -0.7535 
+2024-09-08 07:06:22.169122: Pseudo dice [0.9582, 0.9554, 0.9753, 0.939, 0.8727, 0.9695, 0.9901] 
+2024-09-08 07:06:22.170677: Epoch time: 285.72 s 
+2024-09-08 07:06:24.825240:  
+2024-09-08 07:06:24.826205: Epoch 411 
+2024-09-08 07:06:24.827078: Current learning rate: 0.00621 
+2024-09-08 07:11:05.147049: train_loss -0.7366 
+2024-09-08 07:11:05.148562: val_loss -0.7363 
+2024-09-08 07:11:05.149152: Pseudo dice [0.9657, 0.958, 0.9811, 0.9476, 0.8707, 0.9692, 0.9876] 
+2024-09-08 07:11:05.149610: Epoch time: 280.33 s 
+2024-09-08 07:11:08.449164:  
+2024-09-08 07:11:08.459367: Epoch 412 
+2024-09-08 07:11:08.460382: Current learning rate: 0.0062 
+2024-09-08 07:15:23.899632: train_loss -0.733 
+2024-09-08 07:15:23.902883: val_loss -0.7198 
+2024-09-08 07:15:23.903626: Pseudo dice [0.9575, 0.9559, 0.9757, 0.9438, 0.8687, 0.9689, 0.9912] 
+2024-09-08 07:15:23.922610: Epoch time: 255.46 s 
+2024-09-08 07:15:27.517142:  
+2024-09-08 07:15:27.517950: Epoch 413 
+2024-09-08 07:15:27.518616: Current learning rate: 0.00619 
+2024-09-08 07:20:07.430195: train_loss -0.7366 
+2024-09-08 07:20:07.446144: val_loss -0.7335 
+2024-09-08 07:20:07.446884: Pseudo dice [0.9608, 0.9567, 0.9772, 0.9445, 0.8803, 0.9672, 0.9915] 
+2024-09-08 07:20:07.447557: Epoch time: 279.92 s 
+2024-09-08 07:20:09.685631:  
+2024-09-08 07:20:09.686524: Epoch 414 
+2024-09-08 07:20:09.687622: Current learning rate: 0.00618 
+2024-09-08 07:24:35.586179: train_loss -0.7147 
+2024-09-08 07:24:35.606990: val_loss -0.7408 
+2024-09-08 07:24:35.608306: Pseudo dice [0.9602, 0.9549, 0.975, 0.939, 0.8653, 0.9666, 0.9892] 
+2024-09-08 07:24:35.608960: Epoch time: 265.91 s 
+2024-09-08 07:24:38.224800:  
+2024-09-08 07:24:38.226091: Epoch 415 
+2024-09-08 07:24:38.227743: Current learning rate: 0.00617 
+2024-09-08 07:29:05.084931: train_loss -0.7358 
+2024-09-08 07:29:05.100874: val_loss -0.7579 
+2024-09-08 07:29:05.101579: Pseudo dice [0.9622, 0.9542, 0.9771, 0.9484, 0.8647, 0.9661, 0.9905] 
+2024-09-08 07:29:05.102459: Epoch time: 266.86 s 
+2024-09-08 07:29:09.444978:  
+2024-09-08 07:29:09.445989: Epoch 416 
+2024-09-08 07:29:09.446972: Current learning rate: 0.00616 
+2024-09-08 07:34:07.115925: train_loss -0.7358 
+2024-09-08 07:34:07.117124: val_loss -0.7638 
+2024-09-08 07:34:07.117755: Pseudo dice [0.9611, 0.9558, 0.9802, 0.9521, 0.8789, 0.9684, 0.991] 
+2024-09-08 07:34:07.118278: Epoch time: 297.68 s 
+2024-09-08 07:34:10.820659:  
+2024-09-08 07:34:10.821537: Epoch 417 
+2024-09-08 07:34:10.822822: Current learning rate: 0.00615 
+2024-09-08 07:38:54.197576: train_loss -0.7425 
+2024-09-08 07:38:54.209773: val_loss -0.7382 
+2024-09-08 07:38:54.210826: Pseudo dice [0.9611, 0.9587, 0.9756, 0.9525, 0.8745, 0.9669, 0.9901] 
+2024-09-08 07:38:54.212089: Epoch time: 283.4 s 
+2024-09-08 07:38:58.517443:  
+2024-09-08 07:38:58.518976: Epoch 418 
+2024-09-08 07:38:58.519728: Current learning rate: 0.00614 
+2024-09-08 07:43:21.601232: train_loss -0.7365 
+2024-09-08 07:43:21.603275: val_loss -0.7603 
+2024-09-08 07:43:21.603871: Pseudo dice [0.9626, 0.9576, 0.9783, 0.9507, 0.8571, 0.969, 0.9896] 
+2024-09-08 07:43:21.604456: Epoch time: 263.09 s 
+2024-09-08 07:43:25.037050:  
+2024-09-08 07:43:25.037924: Epoch 419 
+2024-09-08 07:43:25.038460: Current learning rate: 0.00613 
+2024-09-08 07:48:08.599259: train_loss -0.733 
+2024-09-08 07:48:08.601630: val_loss -0.754 
+2024-09-08 07:48:08.602112: Pseudo dice [0.9615, 0.9557, 0.9776, 0.9443, 0.877, 0.9642, 0.9879] 
+2024-09-08 07:48:08.602822: Epoch time: 283.57 s 
+2024-09-08 07:48:12.081840:  
+2024-09-08 07:48:12.082721: Epoch 420 
+2024-09-08 07:48:12.083684: Current learning rate: 0.00612 
+2024-09-08 07:52:52.581078: train_loss -0.727 
+2024-09-08 07:52:52.596547: val_loss -0.7388 
+2024-09-08 07:52:52.597222: Pseudo dice [0.9608, 0.957, 0.9741, 0.9498, 0.8686, 0.9651, 0.9901] 
+2024-09-08 07:52:52.597740: Epoch time: 280.5 s 
+2024-09-08 07:52:55.526819:  
+2024-09-08 07:52:55.528126: Epoch 421 
+2024-09-08 07:52:55.529222: Current learning rate: 0.00612 
+2024-09-08 07:57:35.148510: train_loss -0.747 
+2024-09-08 07:57:35.165871: val_loss -0.735 
+2024-09-08 07:57:35.166712: Pseudo dice [0.9587, 0.9569, 0.9704, 0.9467, 0.8607, 0.9654, 0.9893] 
+2024-09-08 07:57:35.167300: Epoch time: 279.63 s 
+2024-09-08 07:57:38.701868:  
+2024-09-08 07:57:38.702712: Epoch 422 
+2024-09-08 07:57:38.703423: Current learning rate: 0.00611 
+2024-09-08 08:02:15.810580: train_loss -0.7333 
+2024-09-08 08:02:15.812270: val_loss -0.7429 
+2024-09-08 08:02:15.812923: Pseudo dice [0.9573, 0.9522, 0.9738, 0.946, 0.8715, 0.9644, 0.9904] 
+2024-09-08 08:02:15.813479: Epoch time: 277.11 s 
+2024-09-08 08:02:19.090212:  
+2024-09-08 08:02:19.104286: Epoch 423 
+2024-09-08 08:02:19.105099: Current learning rate: 0.0061 
+2024-09-08 08:07:05.076993: train_loss -0.7238 
+2024-09-08 08:07:05.090237: val_loss -0.6932 
+2024-09-08 08:07:05.091145: Pseudo dice [0.9644, 0.9588, 0.9787, 0.9403, 0.8533, 0.9653, 0.9858] 
+2024-09-08 08:07:05.091892: Epoch time: 285.99 s 
+2024-09-08 08:07:08.137350:  
+2024-09-08 08:07:08.152411: Epoch 424 
+2024-09-08 08:07:08.153125: Current learning rate: 0.00609 
+2024-09-08 08:11:57.378681: train_loss -0.7197 
+2024-09-08 08:11:57.393942: val_loss -0.7422 
+2024-09-08 08:11:57.394540: Pseudo dice [0.9613, 0.957, 0.9783, 0.95, 0.8502, 0.9603, 0.9895] 
+2024-09-08 08:11:57.395114: Epoch time: 289.25 s 
+2024-09-08 08:12:01.649635:  
+2024-09-08 08:12:01.650611: Epoch 425 
+2024-09-08 08:12:01.651283: Current learning rate: 0.00608 
+2024-09-08 08:16:36.565814: train_loss -0.7391 
+2024-09-08 08:16:36.581043: val_loss -0.7615 
+2024-09-08 08:16:36.581843: Pseudo dice [0.9648, 0.9583, 0.9795, 0.9515, 0.8586, 0.9695, 0.9908] 
+2024-09-08 08:16:36.582673: Epoch time: 274.92 s 
+2024-09-08 08:16:38.534082:  
+2024-09-08 08:16:38.534970: Epoch 426 
+2024-09-08 08:16:38.535664: Current learning rate: 0.00607 
+2024-09-08 08:21:13.612743: train_loss -0.7404 
+2024-09-08 08:21:13.629608: val_loss -0.7458 
+2024-09-08 08:21:13.630304: Pseudo dice [0.9661, 0.9589, 0.9812, 0.9477, 0.8808, 0.9662, 0.992] 
+2024-09-08 08:21:13.630837: Epoch time: 275.08 s 
+2024-09-08 08:21:16.666022:  
+2024-09-08 08:21:16.667156: Epoch 427 
+2024-09-08 08:21:16.668455: Current learning rate: 0.00606 
+2024-09-08 08:26:02.271393: train_loss -0.7445 
+2024-09-08 08:26:02.284708: val_loss -0.7544 
+2024-09-08 08:26:02.285416: Pseudo dice [0.9643, 0.9582, 0.9799, 0.9478, 0.8789, 0.9716, 0.9903] 
+2024-09-08 08:26:02.285905: Epoch time: 285.63 s 
+2024-09-08 08:26:05.482395:  
+2024-09-08 08:26:05.496532: Epoch 428 
+2024-09-08 08:26:05.497360: Current learning rate: 0.00605 
+2024-09-08 08:30:38.453242: train_loss -0.7371 
+2024-09-08 08:30:38.468756: val_loss -0.7429 
+2024-09-08 08:30:38.469408: Pseudo dice [0.9603, 0.9551, 0.9761, 0.9441, 0.8615, 0.9672, 0.9888] 
+2024-09-08 08:30:38.470107: Epoch time: 272.98 s 
+2024-09-08 08:30:41.704401:  
+2024-09-08 08:30:41.716837: Epoch 429 
+2024-09-08 08:30:41.717662: Current learning rate: 0.00604 
+2024-09-08 08:35:09.136237: train_loss -0.741 
+2024-09-08 08:35:09.138193: val_loss -0.766 
+2024-09-08 08:35:09.138626: Pseudo dice [0.9638, 0.9573, 0.9792, 0.9481, 0.8754, 0.9714, 0.9915] 
+2024-09-08 08:35:09.139398: Epoch time: 267.44 s 
+2024-09-08 08:35:11.167464:  
+2024-09-08 08:35:11.169136: Epoch 430 
+2024-09-08 08:35:11.169874: Current learning rate: 0.00603 
+2024-09-08 08:39:47.998859: train_loss -0.7386 
+2024-09-08 08:39:48.025909: val_loss -0.7423 
+2024-09-08 08:39:48.026582: Pseudo dice [0.966, 0.9604, 0.9803, 0.9516, 0.88, 0.9691, 0.9913] 
+2024-09-08 08:39:48.027131: Epoch time: 276.84 s 
+2024-09-08 08:39:48.027980: Yayy! New best EMA pseudo Dice: 0.9531 
+2024-09-08 08:39:52.897735:  
+2024-09-08 08:39:52.899119: Epoch 431 
+2024-09-08 08:39:52.899678: Current learning rate: 0.00602 
+2024-09-08 08:44:19.176020: train_loss -0.7433 
+2024-09-08 08:44:19.192432: val_loss -0.7594 
+2024-09-08 08:44:19.193481: Pseudo dice [0.964, 0.9574, 0.9798, 0.9485, 0.8761, 0.9635, 0.9915] 
+2024-09-08 08:44:19.194652: Epoch time: 266.28 s 
+2024-09-08 08:44:19.195271: Yayy! New best EMA pseudo Dice: 0.9532 
+2024-09-08 08:44:24.287724:  
+2024-09-08 08:44:24.288585: Epoch 432 
+2024-09-08 08:44:24.289621: Current learning rate: 0.00601 
+2024-09-08 08:48:56.047546: train_loss -0.7462 
+2024-09-08 08:48:56.075791: val_loss -0.7301 
+2024-09-08 08:48:56.077080: Pseudo dice [0.9635, 0.9587, 0.9753, 0.9534, 0.8663, 0.9688, 0.9915] 
+2024-09-08 08:48:56.077764: Epoch time: 271.76 s 
+2024-09-08 08:48:56.078608: Yayy! New best EMA pseudo Dice: 0.9533 
+2024-09-08 08:49:01.617318:  
+2024-09-08 08:49:01.618092: Epoch 433 
+2024-09-08 08:49:01.618860: Current learning rate: 0.006 
+2024-09-08 08:53:34.412666: train_loss -0.7258 
+2024-09-08 08:53:34.427341: val_loss -0.7855 
+2024-09-08 08:53:34.428288: Pseudo dice [0.9652, 0.9588, 0.9798, 0.951, 0.8735, 0.9698, 0.9911] 
+2024-09-08 08:53:34.428846: Epoch time: 272.8 s 
+2024-09-08 08:53:34.429378: Yayy! New best EMA pseudo Dice: 0.9535 
+2024-09-08 08:53:38.877488:  
+2024-09-08 08:53:38.896415: Epoch 434 
+2024-09-08 08:53:38.897361: Current learning rate: 0.00599 
+2024-09-08 08:58:15.673039: train_loss -0.7443 
+2024-09-08 08:58:15.684889: val_loss -0.7288 
+2024-09-08 08:58:15.685495: Pseudo dice [0.962, 0.9553, 0.9777, 0.9477, 0.8595, 0.9634, 0.9868] 
+2024-09-08 08:58:15.686707: Epoch time: 276.8 s 
+2024-09-08 08:58:19.993075:  
+2024-09-08 08:58:19.994365: Epoch 435 
+2024-09-08 08:58:19.995552: Current learning rate: 0.00598 
+2024-09-08 09:03:05.120696: train_loss -0.7456 
+2024-09-08 09:03:05.135335: val_loss -0.7553 
+2024-09-08 09:03:05.135975: Pseudo dice [0.9633, 0.9576, 0.9773, 0.948, 0.874, 0.9672, 0.9893] 
+2024-09-08 09:03:05.136507: Epoch time: 285.13 s 
+2024-09-08 09:03:09.448594:  
+2024-09-08 09:03:09.458641: Epoch 436 
+2024-09-08 09:03:09.459513: Current learning rate: 0.00597 
+2024-09-08 09:08:13.405124: train_loss -0.7269 
+2024-09-08 09:08:13.425016: val_loss -0.7768 
+2024-09-08 09:08:13.425620: Pseudo dice [0.9643, 0.9579, 0.9775, 0.9469, 0.8685, 0.9682, 0.9883] 
+2024-09-08 09:08:13.426120: Epoch time: 303.96 s 
+2024-09-08 09:08:17.274575:  
+2024-09-08 09:08:17.282861: Epoch 437 
+2024-09-08 09:08:17.283921: Current learning rate: 0.00596 
+2024-09-08 09:13:11.738591: train_loss -0.7414 
+2024-09-08 09:13:11.741165: val_loss -0.7061 
+2024-09-08 09:13:11.742334: Pseudo dice [0.9607, 0.9577, 0.9745, 0.9446, 0.8593, 0.9644, 0.9889] 
+2024-09-08 09:13:11.742894: Epoch time: 294.47 s 
+2024-09-08 09:13:14.023639:  
+2024-09-08 09:13:14.025067: Epoch 438 
+2024-09-08 09:13:14.025946: Current learning rate: 0.00595 
+2024-09-08 09:18:11.808816: train_loss -0.7453 
+2024-09-08 09:18:11.828659: val_loss -0.7649 
+2024-09-08 09:18:11.829284: Pseudo dice [0.9633, 0.9576, 0.9778, 0.9468, 0.8723, 0.9713, 0.9901] 
+2024-09-08 09:18:11.829819: Epoch time: 297.79 s 
+2024-09-08 09:18:15.878048:  
+2024-09-08 09:18:15.892728: Epoch 439 
+2024-09-08 09:18:15.893859: Current learning rate: 0.00594 
+2024-09-08 09:22:47.540752: train_loss -0.7321 
+2024-09-08 09:22:47.557376: val_loss -0.7171 
+2024-09-08 09:22:47.557983: Pseudo dice [0.9636, 0.9585, 0.9796, 0.9427, 0.8693, 0.969, 0.988] 
+2024-09-08 09:22:47.558476: Epoch time: 271.67 s 
+2024-09-08 09:22:50.931998:  
+2024-09-08 09:22:50.945727: Epoch 440 
+2024-09-08 09:22:50.946419: Current learning rate: 0.00593 
+2024-09-08 09:27:31.305115: train_loss -0.7259 
+2024-09-08 09:27:31.306319: val_loss -0.7298 
+2024-09-08 09:27:31.306819: Pseudo dice [0.9638, 0.9573, 0.9799, 0.9475, 0.8742, 0.9669, 0.9896] 
+2024-09-08 09:27:31.307284: Epoch time: 280.38 s 
+2024-09-08 09:27:33.360057:  
+2024-09-08 09:27:33.368863: Epoch 441 
+2024-09-08 09:27:33.370065: Current learning rate: 0.00592 
+2024-09-08 09:31:50.235046: train_loss -0.7418 
+2024-09-08 09:31:50.248972: val_loss -0.742 
+2024-09-08 09:31:50.249537: Pseudo dice [0.961, 0.9544, 0.9766, 0.9531, 0.8699, 0.9703, 0.9913] 
+2024-09-08 09:31:50.250056: Epoch time: 256.88 s 
+2024-09-08 09:31:53.677500:  
+2024-09-08 09:31:53.692350: Epoch 442 
+2024-09-08 09:31:53.693120: Current learning rate: 0.00592 
+2024-09-08 09:36:36.349734: train_loss -0.7091 
+2024-09-08 09:36:36.353318: val_loss -0.747 
+2024-09-08 09:36:36.354146: Pseudo dice [0.961, 0.9561, 0.9741, 0.9469, 0.8683, 0.9686, 0.9885] 
+2024-09-08 09:36:36.354878: Epoch time: 282.68 s 
+2024-09-08 09:36:38.898738:  
+2024-09-08 09:36:38.912314: Epoch 443 
+2024-09-08 09:36:38.912998: Current learning rate: 0.00591 
+2024-09-08 09:41:30.879353: train_loss -0.7463 
+2024-09-08 09:41:30.881398: val_loss -0.7807 
+2024-09-08 09:41:30.884803: Pseudo dice [0.963, 0.957, 0.9774, 0.9414, 0.8721, 0.9704, 0.9912] 
+2024-09-08 09:41:30.885758: Epoch time: 291.99 s 
+2024-09-08 09:41:35.480605:  
+2024-09-08 09:41:35.497084: Epoch 444 
+2024-09-08 09:41:35.498196: Current learning rate: 0.0059 
+2024-09-08 09:46:28.111218: train_loss -0.7373 
+2024-09-08 09:46:28.133878: val_loss -0.7297 
+2024-09-08 09:46:28.134509: Pseudo dice [0.9633, 0.956, 0.9794, 0.9428, 0.8661, 0.9707, 0.9906] 
+2024-09-08 09:46:28.135001: Epoch time: 292.63 s 
+2024-09-08 09:46:32.483340:  
+2024-09-08 09:46:32.507755: Epoch 445 
+2024-09-08 09:46:32.516580: Current learning rate: 0.00589 
+2024-09-08 09:51:18.458857: train_loss -0.7477 
+2024-09-08 09:51:18.474284: val_loss -0.7547 
+2024-09-08 09:51:18.475072: Pseudo dice [0.962, 0.9563, 0.975, 0.9437, 0.8746, 0.9616, 0.9904] 
+2024-09-08 09:51:18.476141: Epoch time: 285.98 s 
+2024-09-08 09:51:22.542915:  
+2024-09-08 09:51:22.543994: Epoch 446 
+2024-09-08 09:51:22.544759: Current learning rate: 0.00588 
+2024-09-08 09:57:10.944582: train_loss -0.7182 
+2024-09-08 09:57:10.967875: val_loss -0.7763 
+2024-09-08 09:57:10.968704: Pseudo dice [0.9662, 0.9581, 0.9813, 0.9536, 0.8686, 0.9666, 0.9906] 
+2024-09-08 09:57:10.969384: Epoch time: 348.41 s 
+2024-09-08 09:57:14.840603:  
+2024-09-08 09:57:14.852614: Epoch 447 
+2024-09-08 09:57:14.853506: Current learning rate: 0.00587 
+2024-09-08 10:02:25.189314: train_loss -0.7364 
+2024-09-08 10:02:25.206264: val_loss -0.7327 
+2024-09-08 10:02:25.206967: Pseudo dice [0.9627, 0.9573, 0.9789, 0.9468, 0.8839, 0.9728, 0.9923] 
+2024-09-08 10:02:25.207716: Epoch time: 310.35 s 
+2024-09-08 10:02:29.165924:  
+2024-09-08 10:02:29.166973: Epoch 448 
+2024-09-08 10:02:29.167690: Current learning rate: 0.00586 
+2024-09-08 10:07:23.298169: train_loss -0.7478 
+2024-09-08 10:07:23.308639: val_loss -0.7471 
+2024-09-08 10:07:23.309225: Pseudo dice [0.9637, 0.9588, 0.9754, 0.9473, 0.8813, 0.9708, 0.9912] 
+2024-09-08 10:07:23.309756: Epoch time: 294.14 s 
+2024-09-08 10:07:23.310180: Yayy! New best EMA pseudo Dice: 0.9537 
+2024-09-08 10:07:30.399394:  
+2024-09-08 10:07:30.405236: Epoch 449 
+2024-09-08 10:07:30.405912: Current learning rate: 0.00585 
+2024-09-08 10:12:22.842064: train_loss -0.7315 
+2024-09-08 10:12:22.857921: val_loss -0.7415 
+2024-09-08 10:12:22.858720: Pseudo dice [0.9629, 0.957, 0.9785, 0.9466, 0.869, 0.9613, 0.9904] 
+2024-09-08 10:12:22.859258: Epoch time: 292.45 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_8_14_03_50.txt

@@ -0,0 +1,377 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-08 14:03:54.273034: Using torch.compile... 
+2024-09-08 14:04:00.812350: do_dummy_2d_data_aug: False 
+2024-09-08 14:04:00.817891: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-08 14:04:00.819102: The split file contains 5 splits. 
+2024-09-08 14:04:00.819449: Desired fold for training: 0 
+2024-09-08 14:04:00.819814: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-08 14:04:14.637826: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-08 14:04:15.432646:  
+2024-09-08 14:04:15.448308: Epoch 450 
+2024-09-08 14:04:15.449104: Current learning rate: 0.00584 
+2024-09-08 14:19:21.055386: train_loss -0.729 
+2024-09-08 14:19:21.056510: val_loss -0.7575 
+2024-09-08 14:19:21.057150: Pseudo dice [0.9644, 0.9568, 0.9797, 0.9501, 0.8764, 0.9656, 0.9895] 
+2024-09-08 14:19:21.057648: Epoch time: 905.63 s 
+2024-09-08 14:19:22.966642:  
+2024-09-08 14:19:22.967255: Epoch 451 
+2024-09-08 14:19:22.967729: Current learning rate: 0.00583 
+2024-09-08 14:23:56.220978: train_loss -0.7364 
+2024-09-08 14:23:56.232841: val_loss -0.7659 
+2024-09-08 14:23:56.234119: Pseudo dice [0.9637, 0.9574, 0.9795, 0.9529, 0.8816, 0.967, 0.9906] 
+2024-09-08 14:23:56.235387: Epoch time: 273.26 s 
+2024-09-08 14:23:56.236839: Yayy! New best EMA pseudo Dice: 0.9539 
+2024-09-08 14:24:02.555536:  
+2024-09-08 14:24:02.556917: Epoch 452 
+2024-09-08 14:24:02.557823: Current learning rate: 0.00582 
+2024-09-08 14:28:43.547737: train_loss -0.7445 
+2024-09-08 14:28:43.564761: val_loss -0.7633 
+2024-09-08 14:28:43.566080: Pseudo dice [0.963, 0.9583, 0.9779, 0.9511, 0.8653, 0.9657, 0.9884] 
+2024-09-08 14:28:43.567370: Epoch time: 281.0 s 
+2024-09-08 14:28:46.936069:  
+2024-09-08 14:28:46.937122: Epoch 453 
+2024-09-08 14:28:46.938704: Current learning rate: 0.00581 
+2024-09-08 14:33:15.625608: train_loss -0.7446 
+2024-09-08 14:33:15.642755: val_loss -0.7644 
+2024-09-08 14:33:15.644174: Pseudo dice [0.9627, 0.9581, 0.9783, 0.9542, 0.8911, 0.9631, 0.9922] 
+2024-09-08 14:33:15.646398: Epoch time: 268.69 s 
+2024-09-08 14:33:15.648586: Yayy! New best EMA pseudo Dice: 0.9541 
+2024-09-08 14:33:21.765078:  
+2024-09-08 14:33:21.766016: Epoch 454 
+2024-09-08 14:33:21.767278: Current learning rate: 0.0058 
+2024-09-08 14:38:06.244682: train_loss -0.734 
+2024-09-08 14:38:06.247196: val_loss -0.776 
+2024-09-08 14:38:06.248614: Pseudo dice [0.964, 0.9565, 0.9789, 0.9427, 0.8638, 0.9643, 0.9915] 
+2024-09-08 14:38:06.249601: Epoch time: 284.48 s 
+2024-09-08 14:38:09.352744:  
+2024-09-08 14:38:09.360338: Epoch 455 
+2024-09-08 14:38:09.362347: Current learning rate: 0.00579 
+2024-09-08 14:42:46.768827: train_loss -0.7482 
+2024-09-08 14:42:46.775206: val_loss -0.734 
+2024-09-08 14:42:46.775881: Pseudo dice [0.9622, 0.9589, 0.9758, 0.949, 0.8878, 0.9666, 0.9912] 
+2024-09-08 14:42:46.776459: Epoch time: 277.42 s 
+2024-09-08 14:42:49.901395:  
+2024-09-08 14:42:49.902648: Epoch 456 
+2024-09-08 14:42:49.904244: Current learning rate: 0.00578 
+2024-09-08 14:47:20.604915: train_loss -0.7488 
+2024-09-08 14:47:20.608722: val_loss -0.7434 
+2024-09-08 14:47:20.610051: Pseudo dice [0.964, 0.9582, 0.9783, 0.9504, 0.8749, 0.9695, 0.9902] 
+2024-09-08 14:47:20.611022: Epoch time: 270.71 s 
+2024-09-08 14:47:20.611476: Yayy! New best EMA pseudo Dice: 0.9542 
+2024-09-08 14:47:27.046451:  
+2024-09-08 14:47:27.047326: Epoch 457 
+2024-09-08 14:47:27.047998: Current learning rate: 0.00577 
+2024-09-08 14:51:58.114892: train_loss -0.7331 
+2024-09-08 14:51:58.134455: val_loss -0.7469 
+2024-09-08 14:51:58.135078: Pseudo dice [0.9628, 0.958, 0.9773, 0.9549, 0.8823, 0.9688, 0.9902] 
+2024-09-08 14:51:58.135691: Epoch time: 271.08 s 
+2024-09-08 14:51:58.136132: Yayy! New best EMA pseudo Dice: 0.9544 
+2024-09-08 14:52:03.530777:  
+2024-09-08 14:52:03.536152: Epoch 458 
+2024-09-08 14:52:03.536934: Current learning rate: 0.00576 
+2024-09-08 14:56:39.930537: train_loss -0.7468 
+2024-09-08 14:56:39.944676: val_loss -0.7407 
+2024-09-08 14:56:39.945642: Pseudo dice [0.9655, 0.9588, 0.98, 0.9503, 0.8802, 0.9707, 0.991] 
+2024-09-08 14:56:39.947066: Epoch time: 276.4 s 
+2024-09-08 14:56:39.948561: Yayy! New best EMA pseudo Dice: 0.9546 
+2024-09-08 14:56:45.622419:  
+2024-09-08 14:56:45.625689: Epoch 459 
+2024-09-08 14:56:45.631925: Current learning rate: 0.00575 
+2024-09-08 15:01:14.803575: train_loss -0.7575 
+2024-09-08 15:01:14.805884: val_loss -0.7663 
+2024-09-08 15:01:14.806454: Pseudo dice [0.9664, 0.9581, 0.9804, 0.9582, 0.8767, 0.9718, 0.9911] 
+2024-09-08 15:01:14.806988: Epoch time: 269.19 s 
+2024-09-08 15:01:14.807405: Yayy! New best EMA pseudo Dice: 0.9549 
+2024-09-08 15:01:20.098241:  
+2024-09-08 15:01:20.112271: Epoch 460 
+2024-09-08 15:01:20.113254: Current learning rate: 0.00574 
+2024-09-08 15:05:53.326271: train_loss -0.7451 
+2024-09-08 15:05:53.333956: val_loss -0.749 
+2024-09-08 15:05:53.338461: Pseudo dice [0.9639, 0.9552, 0.9783, 0.9365, 0.8679, 0.9691, 0.9877] 
+2024-09-08 15:05:53.339280: Epoch time: 273.24 s 
+2024-09-08 15:05:56.845785:  
+2024-09-08 15:05:56.847011: Epoch 461 
+2024-09-08 15:05:56.847887: Current learning rate: 0.00573 
+2024-09-08 15:10:43.252251: train_loss -0.7356 
+2024-09-08 15:10:43.269571: val_loss -0.75 
+2024-09-08 15:10:43.270249: Pseudo dice [0.9604, 0.9554, 0.9768, 0.9473, 0.8806, 0.9637, 0.9907] 
+2024-09-08 15:10:43.270705: Epoch time: 286.41 s 
+2024-09-08 15:10:46.929237:  
+2024-09-08 15:10:46.930241: Epoch 462 
+2024-09-08 15:10:46.931373: Current learning rate: 0.00572 
+2024-09-08 15:15:25.175457: train_loss -0.743 
+2024-09-08 15:15:25.205906: val_loss -0.7194 
+2024-09-08 15:15:25.207107: Pseudo dice [0.9623, 0.9578, 0.9766, 0.9507, 0.8564, 0.9618, 0.9886] 
+2024-09-08 15:15:25.207915: Epoch time: 278.28 s 
+2024-09-08 15:15:29.217550:  
+2024-09-08 15:15:29.218421: Epoch 463 
+2024-09-08 15:15:29.218971: Current learning rate: 0.00571 
+2024-09-08 15:19:58.829310: train_loss -0.7397 
+2024-09-08 15:19:58.839213: val_loss -0.7296 
+2024-09-08 15:19:58.839933: Pseudo dice [0.9644, 0.958, 0.9798, 0.9523, 0.8692, 0.9669, 0.9894] 
+2024-09-08 15:19:58.840410: Epoch time: 269.62 s 
+2024-09-08 15:20:02.190151:  
+2024-09-08 15:20:02.204475: Epoch 464 
+2024-09-08 15:20:02.205246: Current learning rate: 0.0057 
+2024-09-08 15:24:55.980764: train_loss -0.7414 
+2024-09-08 15:24:56.001651: val_loss -0.746 
+2024-09-08 15:24:56.002485: Pseudo dice [0.9653, 0.9595, 0.981, 0.949, 0.8794, 0.9706, 0.9904] 
+2024-09-08 15:24:56.003367: Epoch time: 293.8 s 
+2024-09-08 15:25:00.005211:  
+2024-09-08 15:25:00.014069: Epoch 465 
+2024-09-08 15:25:00.014949: Current learning rate: 0.0057 
+2024-09-08 15:29:32.115931: train_loss -0.7477 
+2024-09-08 15:29:32.141378: val_loss -0.7238 
+2024-09-08 15:29:32.142029: Pseudo dice [0.9641, 0.9563, 0.9779, 0.9536, 0.87, 0.9691, 0.9908] 
+2024-09-08 15:29:32.142523: Epoch time: 272.12 s 
+2024-09-08 15:29:35.843097:  
+2024-09-08 15:29:35.844414: Epoch 466 
+2024-09-08 15:29:35.845122: Current learning rate: 0.00569 
+2024-09-08 15:34:31.367184: train_loss -0.747 
+2024-09-08 15:34:31.389351: val_loss -0.7874 
+2024-09-08 15:34:31.390003: Pseudo dice [0.9639, 0.9581, 0.9799, 0.9528, 0.881, 0.9711, 0.9905] 
+2024-09-08 15:34:31.390546: Epoch time: 295.53 s 
+2024-09-08 15:34:36.070287:  
+2024-09-08 15:34:36.071303: Epoch 467 
+2024-09-08 15:34:36.071917: Current learning rate: 0.00568 
+2024-09-08 15:39:21.476404: train_loss -0.7671 
+2024-09-08 15:39:21.478941: val_loss -0.7385 
+2024-09-08 15:39:21.481158: Pseudo dice [0.9632, 0.9573, 0.9789, 0.9501, 0.8693, 0.9678, 0.9906] 
+2024-09-08 15:39:21.483124: Epoch time: 285.41 s 
+2024-09-08 15:39:25.697406:  
+2024-09-08 15:39:25.698460: Epoch 468 
+2024-09-08 15:39:25.699121: Current learning rate: 0.00567 
+2024-09-08 15:44:00.497136: train_loss -0.7403 
+2024-09-08 15:44:00.499443: val_loss -0.751 
+2024-09-08 15:44:00.500064: Pseudo dice [0.9636, 0.9585, 0.9779, 0.9462, 0.8686, 0.9685, 0.9897] 
+2024-09-08 15:44:00.500610: Epoch time: 274.8 s 
+2024-09-08 15:44:03.263258:  
+2024-09-08 15:44:03.264322: Epoch 469 
+2024-09-08 15:44:03.264895: Current learning rate: 0.00566 
+2024-09-08 15:48:35.522401: train_loss -0.7432 
+2024-09-08 15:48:35.536925: val_loss -0.754 
+2024-09-08 15:48:35.537537: Pseudo dice [0.9641, 0.9567, 0.9793, 0.9517, 0.8734, 0.9692, 0.9915] 
+2024-09-08 15:48:35.538066: Epoch time: 272.26 s 
+2024-09-08 15:48:39.221956:  
+2024-09-08 15:48:39.223129: Epoch 470 
+2024-09-08 15:48:39.223932: Current learning rate: 0.00565 
+2024-09-08 15:53:12.586804: train_loss -0.7437 
+2024-09-08 15:53:12.588314: val_loss -0.7458 
+2024-09-08 15:53:12.589282: Pseudo dice [0.9647, 0.9561, 0.977, 0.9516, 0.8748, 0.9613, 0.9912] 
+2024-09-08 15:53:12.590236: Epoch time: 273.37 s 
+2024-09-08 15:53:15.889401:  
+2024-09-08 15:53:15.890385: Epoch 471 
+2024-09-08 15:53:15.891900: Current learning rate: 0.00564 
+2024-09-08 15:57:55.593725: train_loss -0.7554 
+2024-09-08 15:57:55.596084: val_loss -0.7409 
+2024-09-08 15:57:55.597616: Pseudo dice [0.9635, 0.9578, 0.9799, 0.9455, 0.8808, 0.9694, 0.9902] 
+2024-09-08 15:57:55.598948: Epoch time: 279.71 s 
+2024-09-08 15:57:57.922307:  
+2024-09-08 15:57:57.948412: Epoch 472 
+2024-09-08 15:57:57.949350: Current learning rate: 0.00563 
+2024-09-08 16:02:40.056642: train_loss -0.7385 
+2024-09-08 16:02:40.059980: val_loss -0.7415 
+2024-09-08 16:02:40.060819: Pseudo dice [0.9635, 0.9569, 0.9748, 0.9475, 0.8771, 0.9643, 0.9902] 
+2024-09-08 16:02:40.061575: Epoch time: 282.14 s 
+2024-09-08 16:02:43.785878:  
+2024-09-08 16:02:43.787081: Epoch 473 
+2024-09-08 16:02:43.787759: Current learning rate: 0.00562 
+2024-09-08 16:07:38.200438: train_loss -0.7291 
+2024-09-08 16:07:38.219869: val_loss -0.7305 
+2024-09-08 16:07:38.221150: Pseudo dice [0.9629, 0.9578, 0.9753, 0.9433, 0.879, 0.9677, 0.9914] 
+2024-09-08 16:07:38.222874: Epoch time: 294.42 s 
+2024-09-08 16:07:42.857609:  
+2024-09-08 16:07:42.872407: Epoch 474 
+2024-09-08 16:07:42.873230: Current learning rate: 0.00561 
+2024-09-08 16:12:32.885076: train_loss -0.7518 
+2024-09-08 16:12:32.910330: val_loss -0.7658 
+2024-09-08 16:12:32.911003: Pseudo dice [0.9637, 0.9572, 0.9779, 0.9499, 0.8671, 0.9675, 0.9879] 
+2024-09-08 16:12:32.911712: Epoch time: 290.03 s 
+2024-09-08 16:12:37.048969:  
+2024-09-08 16:12:37.060367: Epoch 475 
+2024-09-08 16:12:37.061133: Current learning rate: 0.0056 
+2024-09-08 16:17:18.160405: train_loss -0.745 
+2024-09-08 16:17:18.168730: val_loss -0.7366 
+2024-09-08 16:17:18.169273: Pseudo dice [0.9652, 0.9579, 0.9787, 0.9477, 0.8763, 0.9685, 0.9895] 
+2024-09-08 16:17:18.169785: Epoch time: 281.12 s 
+2024-09-08 16:17:20.886443:  
+2024-09-08 16:17:20.887330: Epoch 476 
+2024-09-08 16:17:20.888005: Current learning rate: 0.00559 
+2024-09-08 16:21:49.571983: train_loss -0.7389 
+2024-09-08 16:21:49.584606: val_loss -0.7072 
+2024-09-08 16:21:49.585277: Pseudo dice [0.9605, 0.9571, 0.9748, 0.9514, 0.8721, 0.9707, 0.9897] 
+2024-09-08 16:21:49.585841: Epoch time: 268.69 s 
+2024-09-08 16:21:54.276763:  
+2024-09-08 16:21:54.288524: Epoch 477 
+2024-09-08 16:21:54.289860: Current learning rate: 0.00558 
+2024-09-08 16:26:26.635769: train_loss -0.7234 
+2024-09-08 16:26:26.643193: val_loss -0.7284 
+2024-09-08 16:26:26.643939: Pseudo dice [0.9604, 0.955, 0.976, 0.9408, 0.8644, 0.9604, 0.9903] 
+2024-09-08 16:26:26.644573: Epoch time: 272.36 s 
+2024-09-08 16:26:30.024885:  
+2024-09-08 16:26:30.025684: Epoch 478 
+2024-09-08 16:26:30.026194: Current learning rate: 0.00557 
+2024-09-08 16:31:25.987343: train_loss -0.7155 
+2024-09-08 16:31:26.001643: val_loss -0.7432 
+2024-09-08 16:31:26.002565: Pseudo dice [0.9617, 0.9553, 0.9753, 0.9442, 0.8661, 0.9674, 0.9889] 
+2024-09-08 16:31:26.003067: Epoch time: 295.97 s 
+2024-09-08 16:31:29.404931:  
+2024-09-08 16:31:29.410775: Epoch 479 
+2024-09-08 16:31:29.411551: Current learning rate: 0.00556 
+2024-09-08 16:36:14.983012: train_loss -0.7494 
+2024-09-08 16:36:14.984317: val_loss -0.773 
+2024-09-08 16:36:14.984837: Pseudo dice [0.966, 0.9591, 0.9794, 0.9505, 0.877, 0.9698, 0.9909] 
+2024-09-08 16:36:14.985323: Epoch time: 285.58 s 
+2024-09-08 16:36:18.768875:  
+2024-09-08 16:36:18.769656: Epoch 480 
+2024-09-08 16:36:18.770175: Current learning rate: 0.00555 
+2024-09-08 16:40:53.388192: train_loss -0.7246 
+2024-09-08 16:40:53.395907: val_loss -0.721 
+2024-09-08 16:40:53.396740: Pseudo dice [0.9623, 0.956, 0.9726, 0.9471, 0.8743, 0.9671, 0.987] 
+2024-09-08 16:40:53.397686: Epoch time: 274.62 s 
+2024-09-08 16:41:00.589523:  
+2024-09-08 16:41:00.604748: Epoch 481 
+2024-09-08 16:41:00.605890: Current learning rate: 0.00554 
+2024-09-08 16:45:48.452660: train_loss -0.7505 
+2024-09-08 16:45:48.472543: val_loss -0.7647 
+2024-09-08 16:45:48.473340: Pseudo dice [0.9639, 0.9572, 0.9793, 0.951, 0.8753, 0.9669, 0.9898] 
+2024-09-08 16:45:48.474020: Epoch time: 287.87 s 
+2024-09-08 16:45:50.852355:  
+2024-09-08 16:45:50.865699: Epoch 482 
+2024-09-08 16:45:50.866596: Current learning rate: 0.00553 
+2024-09-08 16:50:31.511200: train_loss -0.7407 
+2024-09-08 16:50:31.524805: val_loss -0.733 
+2024-09-08 16:50:31.525574: Pseudo dice [0.9642, 0.9607, 0.9784, 0.9513, 0.8851, 0.9671, 0.9899] 
+2024-09-08 16:50:31.526270: Epoch time: 280.66 s 
+2024-09-08 16:50:34.909134:  
+2024-09-08 16:50:34.916449: Epoch 483 
+2024-09-08 16:50:34.917330: Current learning rate: 0.00552 
+2024-09-08 16:55:19.692008: train_loss -0.7429 
+2024-09-08 16:55:19.697055: val_loss -0.7558 
+2024-09-08 16:55:19.697802: Pseudo dice [0.9648, 0.9577, 0.9806, 0.9475, 0.8803, 0.9718, 0.9918] 
+2024-09-08 16:55:19.698457: Epoch time: 284.8 s 
+2024-09-08 16:55:24.267371:  
+2024-09-08 16:55:24.268308: Epoch 484 
+2024-09-08 16:55:24.269118: Current learning rate: 0.00551 
+2024-09-08 17:00:19.173885: train_loss -0.7353 
+2024-09-08 17:00:19.191831: val_loss -0.7314 
+2024-09-08 17:00:19.192655: Pseudo dice [0.9651, 0.9563, 0.9798, 0.9481, 0.8754, 0.969, 0.9923] 
+2024-09-08 17:00:19.193178: Epoch time: 294.91 s 
+2024-09-08 17:00:22.829947:  
+2024-09-08 17:00:22.830877: Epoch 485 
+2024-09-08 17:00:22.831615: Current learning rate: 0.0055 
+2024-09-08 17:04:52.509879: train_loss -0.7482 
+2024-09-08 17:04:52.521424: val_loss -0.7194 
+2024-09-08 17:04:52.522354: Pseudo dice [0.9626, 0.9606, 0.9761, 0.9512, 0.8726, 0.9678, 0.9907] 
+2024-09-08 17:04:52.522909: Epoch time: 269.69 s 
+2024-09-08 17:04:56.827775:  
+2024-09-08 17:04:56.852458: Epoch 486 
+2024-09-08 17:04:56.853212: Current learning rate: 0.00549 
+2024-09-08 17:09:43.398751: train_loss -0.7411 
+2024-09-08 17:09:43.412667: val_loss -0.7377 
+2024-09-08 17:09:43.413243: Pseudo dice [0.9592, 0.9545, 0.9762, 0.9477, 0.8708, 0.9681, 0.9892] 
+2024-09-08 17:09:43.413745: Epoch time: 286.58 s 
+2024-09-08 17:09:47.497977:  
+2024-09-08 17:09:47.512257: Epoch 487 
+2024-09-08 17:09:47.513063: Current learning rate: 0.00548 
+2024-09-08 17:14:37.665298: train_loss -0.7496 
+2024-09-08 17:14:37.685325: val_loss -0.7761 
+2024-09-08 17:14:37.685966: Pseudo dice [0.9643, 0.9576, 0.9777, 0.9475, 0.8694, 0.9718, 0.9892] 
+2024-09-08 17:14:37.686655: Epoch time: 290.17 s 
+2024-09-08 17:14:41.808928:  
+2024-09-08 17:14:41.820244: Epoch 488 
+2024-09-08 17:14:41.820826: Current learning rate: 0.00547 
+2024-09-08 17:19:33.705006: train_loss -0.7168 
+2024-09-08 17:19:33.707323: val_loss -0.6909 
+2024-09-08 17:19:33.707994: Pseudo dice [0.9602, 0.9537, 0.9736, 0.9381, 0.859, 0.9597, 0.9837] 
+2024-09-08 17:19:33.709045: Epoch time: 291.9 s 
+2024-09-08 17:19:38.073471:  
+2024-09-08 17:19:38.084363: Epoch 489 
+2024-09-08 17:19:38.086061: Current learning rate: 0.00546 
+2024-09-08 17:24:19.430495: train_loss -0.7143 
+2024-09-08 17:24:19.432654: val_loss -0.7146 
+2024-09-08 17:24:19.433167: Pseudo dice [0.9591, 0.9546, 0.9706, 0.9413, 0.859, 0.9665, 0.9911] 
+2024-09-08 17:24:19.433730: Epoch time: 281.36 s 
+2024-09-08 17:24:23.686472:  
+2024-09-08 17:24:23.700433: Epoch 490 
+2024-09-08 17:24:23.701468: Current learning rate: 0.00546 
+2024-09-08 17:29:12.087641: train_loss -0.7267 
+2024-09-08 17:29:12.102175: val_loss -0.7364 
+2024-09-08 17:29:12.102848: Pseudo dice [0.9636, 0.9572, 0.9763, 0.9481, 0.8701, 0.9659, 0.9867] 
+2024-09-08 17:29:12.103354: Epoch time: 288.41 s 
+2024-09-08 17:29:16.392080:  
+2024-09-08 17:29:16.408401: Epoch 491 
+2024-09-08 17:29:16.409109: Current learning rate: 0.00545 
+2024-09-08 17:34:05.145178: train_loss -0.7127 
+2024-09-08 17:34:05.156855: val_loss -0.746 
+2024-09-08 17:34:05.158473: Pseudo dice [0.9568, 0.9508, 0.9748, 0.9336, 0.8547, 0.9572, 0.989] 
+2024-09-08 17:34:05.159881: Epoch time: 288.76 s 
+2024-09-08 17:34:09.269094:  
+2024-09-08 17:34:09.270730: Epoch 492 
+2024-09-08 17:34:09.272146: Current learning rate: 0.00544 
+2024-09-08 17:38:44.515565: train_loss -0.7265 
+2024-09-08 17:38:44.533928: val_loss -0.7307 
+2024-09-08 17:38:44.534496: Pseudo dice [0.9606, 0.9536, 0.9769, 0.9381, 0.8623, 0.9613, 0.9899] 
+2024-09-08 17:38:44.534989: Epoch time: 275.25 s 
+2024-09-08 17:38:48.620709:  
+2024-09-08 17:38:48.621719: Epoch 493 
+2024-09-08 17:38:48.622262: Current learning rate: 0.00543 
+2024-09-08 17:43:21.038129: train_loss -0.7207 
+2024-09-08 17:43:21.040355: val_loss -0.7683 
+2024-09-08 17:43:21.041092: Pseudo dice [0.9626, 0.955, 0.9759, 0.9484, 0.8672, 0.9693, 0.9916] 
+2024-09-08 17:43:21.041804: Epoch time: 272.42 s 
+2024-09-08 17:43:24.900459:  
+2024-09-08 17:43:24.901376: Epoch 494 
+2024-09-08 17:43:24.902142: Current learning rate: 0.00542 
+2024-09-08 17:48:20.685215: train_loss -0.7517 
+2024-09-08 17:48:20.715850: val_loss -0.748 
+2024-09-08 17:48:20.716600: Pseudo dice [0.9633, 0.9581, 0.9784, 0.9481, 0.8911, 0.9702, 0.9916] 
+2024-09-08 17:48:20.717116: Epoch time: 295.79 s 
+2024-09-08 17:48:25.174015:  
+2024-09-08 17:48:25.175609: Epoch 495 
+2024-09-08 17:48:25.177071: Current learning rate: 0.00541 
+2024-09-08 17:52:59.490682: train_loss -0.7397 
+2024-09-08 17:52:59.511573: val_loss -0.7427 
+2024-09-08 17:52:59.512279: Pseudo dice [0.9634, 0.9563, 0.9786, 0.9489, 0.8753, 0.9672, 0.9905] 
+2024-09-08 17:52:59.513068: Epoch time: 274.32 s 
+2024-09-08 17:53:03.957477:  
+2024-09-08 17:53:03.961877: Epoch 496 
+2024-09-08 17:53:03.962720: Current learning rate: 0.0054 
+2024-09-08 17:57:40.447337: train_loss -0.7366 
+2024-09-08 17:57:40.449561: val_loss -0.7436 
+2024-09-08 17:57:40.450066: Pseudo dice [0.9651, 0.9591, 0.9779, 0.9477, 0.877, 0.9725, 0.9909] 
+2024-09-08 17:57:40.450645: Epoch time: 276.49 s 
+2024-09-08 17:57:44.504834:  
+2024-09-08 17:57:44.516523: Epoch 497 
+2024-09-08 17:57:44.517441: Current learning rate: 0.00539 
+2024-09-08 18:02:30.814933: train_loss -0.7359 
+2024-09-08 18:02:30.827422: val_loss -0.7497 
+2024-09-08 18:02:30.828358: Pseudo dice [0.9635, 0.956, 0.9799, 0.9495, 0.8774, 0.9645, 0.9915] 
+2024-09-08 18:02:30.829067: Epoch time: 286.32 s 
+2024-09-08 18:02:34.890568:  
+2024-09-08 18:02:34.891672: Epoch 498 
+2024-09-08 18:02:34.893206: Current learning rate: 0.00538 
+2024-09-08 18:06:51.609505: train_loss -0.7106 
+2024-09-08 18:06:51.632809: val_loss -0.6811 
+2024-09-08 18:06:51.633418: Pseudo dice [0.9583, 0.9556, 0.9717, 0.9371, 0.8567, 0.9608, 0.9867] 
+2024-09-08 18:06:51.633897: Epoch time: 256.73 s 
+2024-09-08 18:06:55.066619:  
+2024-09-08 18:06:55.076257: Epoch 499 
+2024-09-08 18:06:55.076997: Current learning rate: 0.00537 
+2024-09-08 18:11:54.942662: train_loss -0.71 
+2024-09-08 18:11:54.956793: val_loss -0.7029 
+2024-09-08 18:11:54.957465: Pseudo dice [0.9556, 0.9479, 0.9719, 0.9325, 0.8461, 0.9529, 0.988] 
+2024-09-08 18:11:54.958055: Epoch time: 299.88 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_8_22_03_23.txt

@@ -0,0 +1,372 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-08 22:03:26.611473: Using torch.compile... 
+2024-09-08 22:03:33.033154: do_dummy_2d_data_aug: False 
+2024-09-08 22:03:33.045603: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-08 22:03:33.046647: The split file contains 5 splits. 
+2024-09-08 22:03:33.047061: Desired fold for training: 0 
+2024-09-08 22:03:33.047448: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-08 22:03:56.420992: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-08 22:03:57.952710:  
+2024-09-08 22:03:57.964610: Epoch 500 
+2024-09-08 22:03:57.965943: Current learning rate: 0.00536 
+2024-09-08 22:19:27.422375: train_loss -0.7028 
+2024-09-08 22:19:27.424257: val_loss -0.7499 
+2024-09-08 22:19:27.424704: Pseudo dice [0.9565, 0.9497, 0.9719, 0.94, 0.8496, 0.9619, 0.9893] 
+2024-09-08 22:19:27.425285: Epoch time: 929.48 s 
+2024-09-08 22:19:29.612756:  
+2024-09-08 22:19:29.614308: Epoch 501 
+2024-09-08 22:19:29.615089: Current learning rate: 0.00535 
+2024-09-08 22:26:26.622248: train_loss -0.7332 
+2024-09-08 22:26:26.634089: val_loss -0.7213 
+2024-09-08 22:26:26.636500: Pseudo dice [0.9607, 0.953, 0.9773, 0.9425, 0.8495, 0.9603, 0.9902] 
+2024-09-08 22:26:26.638394: Epoch time: 417.04 s 
+2024-09-08 22:26:30.912707:  
+2024-09-08 22:26:30.939817: Epoch 502 
+2024-09-08 22:26:30.941099: Current learning rate: 0.00534 
+2024-09-08 22:33:41.010464: train_loss -0.7423 
+2024-09-08 22:33:41.057413: val_loss -0.7489 
+2024-09-08 22:33:41.058676: Pseudo dice [0.9621, 0.9582, 0.9746, 0.9483, 0.8715, 0.9684, 0.9886] 
+2024-09-08 22:33:41.059583: Epoch time: 430.1 s 
+2024-09-08 22:33:45.717312:  
+2024-09-08 22:33:45.740955: Epoch 503 
+2024-09-08 22:33:45.742644: Current learning rate: 0.00533 
+2024-09-08 22:40:55.652017: train_loss -0.7326 
+2024-09-08 22:40:55.669107: val_loss -0.7486 
+2024-09-08 22:40:55.669807: Pseudo dice [0.9603, 0.9546, 0.9748, 0.9484, 0.8694, 0.9687, 0.9885] 
+2024-09-08 22:40:55.670464: Epoch time: 429.95 s 
+2024-09-08 22:41:01.601744:  
+2024-09-08 22:41:01.603212: Epoch 504 
+2024-09-08 22:41:01.604028: Current learning rate: 0.00532 
+2024-09-08 22:48:04.672549: train_loss -0.7333 
+2024-09-08 22:48:04.675196: val_loss -0.7604 
+2024-09-08 22:48:04.675842: Pseudo dice [0.9643, 0.9561, 0.9785, 0.9446, 0.8565, 0.9662, 0.9883] 
+2024-09-08 22:48:04.676446: Epoch time: 423.08 s 
+2024-09-08 22:48:08.910458:  
+2024-09-08 22:48:08.911749: Epoch 505 
+2024-09-08 22:48:08.913869: Current learning rate: 0.00531 
+2024-09-08 22:55:06.415637: train_loss -0.7346 
+2024-09-08 22:55:06.429293: val_loss -0.7515 
+2024-09-08 22:55:06.430404: Pseudo dice [0.963, 0.9564, 0.9782, 0.9478, 0.8616, 0.9632, 0.9888] 
+2024-09-08 22:55:06.431367: Epoch time: 417.51 s 
+2024-09-08 22:55:11.279435:  
+2024-09-08 22:55:11.280894: Epoch 506 
+2024-09-08 22:55:11.293131: Current learning rate: 0.0053 
+2024-09-08 23:01:53.388918: train_loss -0.7486 
+2024-09-08 23:01:53.393240: val_loss -0.7576 
+2024-09-08 23:01:53.393996: Pseudo dice [0.9629, 0.9581, 0.9798, 0.9386, 0.8818, 0.9675, 0.9914] 
+2024-09-08 23:01:53.395457: Epoch time: 402.15 s 
+2024-09-08 23:01:58.895018:  
+2024-09-08 23:01:58.909301: Epoch 507 
+2024-09-08 23:01:58.910817: Current learning rate: 0.00529 
+2024-09-08 23:09:07.472039: train_loss -0.7394 
+2024-09-08 23:09:07.482513: val_loss -0.7296 
+2024-09-08 23:09:07.483279: Pseudo dice [0.9599, 0.9541, 0.9771, 0.9414, 0.8366, 0.964, 0.9859] 
+2024-09-08 23:09:07.483835: Epoch time: 428.62 s 
+2024-09-08 23:09:11.802317:  
+2024-09-08 23:09:11.824444: Epoch 508 
+2024-09-08 23:09:11.825875: Current learning rate: 0.00528 
+2024-09-08 23:16:11.654880: train_loss -0.7199 
+2024-09-08 23:16:11.666627: val_loss -0.7716 
+2024-09-08 23:16:11.667734: Pseudo dice [0.9601, 0.954, 0.9719, 0.9447, 0.8728, 0.9685, 0.9918] 
+2024-09-08 23:16:11.668591: Epoch time: 419.86 s 
+2024-09-08 23:16:16.208230:  
+2024-09-08 23:16:16.225000: Epoch 509 
+2024-09-08 23:16:16.226320: Current learning rate: 0.00527 
+2024-09-08 23:23:11.642759: train_loss -0.7252 
+2024-09-08 23:23:11.644531: val_loss -0.7344 
+2024-09-08 23:23:11.645201: Pseudo dice [0.9618, 0.9555, 0.9762, 0.9474, 0.8724, 0.9704, 0.9905] 
+2024-09-08 23:23:11.645857: Epoch time: 415.46 s 
+2024-09-08 23:23:16.833740:  
+2024-09-08 23:23:16.862376: Epoch 510 
+2024-09-08 23:23:16.863755: Current learning rate: 0.00526 
+2024-09-08 23:30:30.379908: train_loss -0.7237 
+2024-09-08 23:30:30.384095: val_loss -0.7065 
+2024-09-08 23:30:30.386073: Pseudo dice [0.9605, 0.9574, 0.9753, 0.9469, 0.8718, 0.9676, 0.9909] 
+2024-09-08 23:30:30.386997: Epoch time: 433.56 s 
+2024-09-08 23:30:35.647508:  
+2024-09-08 23:30:35.650318: Epoch 511 
+2024-09-08 23:30:35.653536: Current learning rate: 0.00525 
+2024-09-08 23:37:47.536136: train_loss -0.7231 
+2024-09-08 23:37:47.546269: val_loss -0.7245 
+2024-09-08 23:37:47.547252: Pseudo dice [0.9607, 0.9539, 0.975, 0.9384, 0.8453, 0.964, 0.9864] 
+2024-09-08 23:37:47.548748: Epoch time: 431.9 s 
+2024-09-08 23:37:51.277704:  
+2024-09-08 23:37:51.279005: Epoch 512 
+2024-09-08 23:37:51.279802: Current learning rate: 0.00524 
+2024-09-08 23:44:41.059899: train_loss -0.7425 
+2024-09-08 23:44:41.091231: val_loss -0.7454 
+2024-09-08 23:44:41.092193: Pseudo dice [0.9633, 0.9563, 0.9793, 0.9545, 0.8712, 0.9698, 0.9822] 
+2024-09-08 23:44:41.093141: Epoch time: 409.79 s 
+2024-09-08 23:44:46.925447:  
+2024-09-08 23:44:46.947827: Epoch 513 
+2024-09-08 23:44:46.948973: Current learning rate: 0.00523 
+2024-09-08 23:52:07.988736: train_loss -0.7325 
+2024-09-08 23:52:07.990494: val_loss -0.7151 
+2024-09-08 23:52:07.992243: Pseudo dice [0.9628, 0.9556, 0.9791, 0.9478, 0.8706, 0.9665, 0.9905] 
+2024-09-08 23:52:07.993863: Epoch time: 441.07 s 
+2024-09-08 23:52:13.001881:  
+2024-09-08 23:52:13.003160: Epoch 514 
+2024-09-08 23:52:13.004694: Current learning rate: 0.00522 
+2024-09-08 23:59:15.486805: train_loss -0.7415 
+2024-09-08 23:59:15.498006: val_loss -0.7484 
+2024-09-08 23:59:15.498791: Pseudo dice [0.9627, 0.9578, 0.9798, 0.9495, 0.8665, 0.9615, 0.9909] 
+2024-09-08 23:59:15.499434: Epoch time: 422.49 s 
+2024-09-08 23:59:22.867746:  
+2024-09-08 23:59:22.869993: Epoch 515 
+2024-09-08 23:59:22.871035: Current learning rate: 0.00521 
+2024-09-09 00:06:36.349681: train_loss -0.7198 
+2024-09-09 00:06:36.377197: val_loss -0.759 
+2024-09-09 00:06:36.378409: Pseudo dice [0.9591, 0.9548, 0.9714, 0.9442, 0.8573, 0.9643, 0.9897] 
+2024-09-09 00:06:36.379909: Epoch time: 433.49 s 
+2024-09-09 00:06:40.498843:  
+2024-09-09 00:06:40.500320: Epoch 516 
+2024-09-09 00:06:40.501693: Current learning rate: 0.0052 
+2024-09-09 00:13:48.139335: train_loss -0.7372 
+2024-09-09 00:13:48.148175: val_loss -0.7396 
+2024-09-09 00:13:48.149760: Pseudo dice [0.963, 0.9571, 0.9762, 0.9429, 0.8771, 0.9696, 0.9901] 
+2024-09-09 00:13:48.150892: Epoch time: 427.65 s 
+2024-09-09 00:13:52.609016:  
+2024-09-09 00:13:52.615847: Epoch 517 
+2024-09-09 00:13:52.617053: Current learning rate: 0.00519 
+2024-09-09 00:21:04.958247: train_loss -0.7399 
+2024-09-09 00:21:04.961122: val_loss -0.7303 
+2024-09-09 00:21:04.963357: Pseudo dice [0.9663, 0.9602, 0.9818, 0.9515, 0.8749, 0.9644, 0.9919] 
+2024-09-09 00:21:04.965775: Epoch time: 432.36 s 
+2024-09-09 00:21:09.383941:  
+2024-09-09 00:21:09.396617: Epoch 518 
+2024-09-09 00:21:09.398214: Current learning rate: 0.00518 
+2024-09-09 00:28:24.910494: train_loss -0.7435 
+2024-09-09 00:28:24.934504: val_loss -0.7518 
+2024-09-09 00:28:24.935480: Pseudo dice [0.962, 0.955, 0.9762, 0.9506, 0.8663, 0.9693, 0.9897] 
+2024-09-09 00:28:24.936373: Epoch time: 435.53 s 
+2024-09-09 00:28:28.414535:  
+2024-09-09 00:28:28.415797: Epoch 519 
+2024-09-09 00:28:28.416475: Current learning rate: 0.00518 
+2024-09-09 00:35:09.133358: train_loss -0.737 
+2024-09-09 00:35:09.155640: val_loss -0.7346 
+2024-09-09 00:35:09.157016: Pseudo dice [0.9601, 0.9575, 0.9765, 0.9527, 0.8622, 0.9694, 0.9898] 
+2024-09-09 00:35:09.158292: Epoch time: 400.72 s 
+2024-09-09 00:35:14.151243:  
+2024-09-09 00:35:14.152608: Epoch 520 
+2024-09-09 00:35:14.154913: Current learning rate: 0.00517 
+2024-09-09 00:42:37.157781: train_loss -0.7446 
+2024-09-09 00:42:37.179786: val_loss -0.7528 
+2024-09-09 00:42:37.182991: Pseudo dice [0.9619, 0.9556, 0.9752, 0.9445, 0.8681, 0.9693, 0.9912] 
+2024-09-09 00:42:37.185348: Epoch time: 443.01 s 
+2024-09-09 00:42:41.419550:  
+2024-09-09 00:42:41.436176: Epoch 521 
+2024-09-09 00:42:41.437085: Current learning rate: 0.00516 
+2024-09-09 00:49:46.179238: train_loss -0.753 
+2024-09-09 00:49:46.187469: val_loss -0.7412 
+2024-09-09 00:49:46.188428: Pseudo dice [0.9642, 0.9565, 0.9788, 0.9492, 0.8797, 0.9681, 0.9917] 
+2024-09-09 00:49:46.189920: Epoch time: 424.77 s 
+2024-09-09 00:49:51.086575:  
+2024-09-09 00:49:51.087970: Epoch 522 
+2024-09-09 00:49:51.089629: Current learning rate: 0.00515 
+2024-09-09 00:56:44.551215: train_loss -0.7494 
+2024-09-09 00:56:44.574269: val_loss -0.7298 
+2024-09-09 00:56:44.576070: Pseudo dice [0.9641, 0.9566, 0.9773, 0.952, 0.8813, 0.9695, 0.9901] 
+2024-09-09 00:56:44.578560: Epoch time: 413.46 s 
+2024-09-09 00:56:49.379161:  
+2024-09-09 00:56:49.380402: Epoch 523 
+2024-09-09 00:56:49.381860: Current learning rate: 0.00514 
+2024-09-09 01:04:00.100420: train_loss -0.752 
+2024-09-09 01:04:00.105503: val_loss -0.7653 
+2024-09-09 01:04:00.107009: Pseudo dice [0.9628, 0.9563, 0.9779, 0.9465, 0.8793, 0.9703, 0.9907] 
+2024-09-09 01:04:00.108047: Epoch time: 430.73 s 
+2024-09-09 01:04:05.637643:  
+2024-09-09 01:04:05.667927: Epoch 524 
+2024-09-09 01:04:05.675185: Current learning rate: 0.00513 
+2024-09-09 01:10:57.912194: train_loss -0.7427 
+2024-09-09 01:10:57.926602: val_loss -0.7518 
+2024-09-09 01:10:57.927423: Pseudo dice [0.9631, 0.9575, 0.9775, 0.9488, 0.8826, 0.9683, 0.991] 
+2024-09-09 01:10:57.928049: Epoch time: 412.28 s 
+2024-09-09 01:11:02.332352:  
+2024-09-09 01:11:02.348680: Epoch 525 
+2024-09-09 01:11:02.349631: Current learning rate: 0.00512 
+2024-09-09 01:18:18.272375: train_loss -0.7462 
+2024-09-09 01:18:18.294726: val_loss -0.75 
+2024-09-09 01:18:18.296029: Pseudo dice [0.9627, 0.9564, 0.9777, 0.9441, 0.8732, 0.9693, 0.9911] 
+2024-09-09 01:18:18.297350: Epoch time: 435.96 s 
+2024-09-09 01:18:23.093782:  
+2024-09-09 01:18:23.094761: Epoch 526 
+2024-09-09 01:18:23.095388: Current learning rate: 0.00511 
+2024-09-09 01:25:48.080901: train_loss -0.7411 
+2024-09-09 01:25:48.100637: val_loss -0.7477 
+2024-09-09 01:25:48.102691: Pseudo dice [0.9607, 0.9572, 0.9783, 0.9473, 0.8608, 0.9637, 0.9909] 
+2024-09-09 01:25:48.104345: Epoch time: 444.99 s 
+2024-09-09 01:25:52.252002:  
+2024-09-09 01:25:52.254082: Epoch 527 
+2024-09-09 01:25:52.256802: Current learning rate: 0.0051 
+2024-09-09 01:32:32.252851: train_loss -0.7356 
+2024-09-09 01:32:32.270261: val_loss -0.7719 
+2024-09-09 01:32:32.271268: Pseudo dice [0.9626, 0.9573, 0.9773, 0.9489, 0.8751, 0.9691, 0.9908] 
+2024-09-09 01:32:32.273330: Epoch time: 400.06 s 
+2024-09-09 01:32:36.966910:  
+2024-09-09 01:32:36.968307: Epoch 528 
+2024-09-09 01:32:36.969315: Current learning rate: 0.00509 
+2024-09-09 01:39:23.048784: train_loss -0.7274 
+2024-09-09 01:39:23.050275: val_loss -0.7668 
+2024-09-09 01:39:23.051062: Pseudo dice [0.9635, 0.9557, 0.9786, 0.9465, 0.8759, 0.9681, 0.9919] 
+2024-09-09 01:39:23.051639: Epoch time: 406.12 s 
+2024-09-09 01:39:31.634104:  
+2024-09-09 01:39:31.649540: Epoch 529 
+2024-09-09 01:39:31.650859: Current learning rate: 0.00508 
+2024-09-09 01:46:42.835015: train_loss -0.7415 
+2024-09-09 01:46:42.840488: val_loss -0.7056 
+2024-09-09 01:46:42.841342: Pseudo dice [0.9633, 0.9588, 0.9771, 0.9508, 0.8792, 0.9632, 0.9917] 
+2024-09-09 01:46:42.842641: Epoch time: 431.21 s 
+2024-09-09 01:46:46.792964:  
+2024-09-09 01:46:46.800486: Epoch 530 
+2024-09-09 01:46:46.801373: Current learning rate: 0.00507 
+2024-09-09 01:53:31.901354: train_loss -0.7337 
+2024-09-09 01:53:31.908507: val_loss -0.7498 
+2024-09-09 01:53:31.917368: Pseudo dice [0.9645, 0.9583, 0.979, 0.9502, 0.8714, 0.9656, 0.9907] 
+2024-09-09 01:53:31.918660: Epoch time: 405.11 s 
+2024-09-09 01:53:36.427330:  
+2024-09-09 01:53:36.428402: Epoch 531 
+2024-09-09 01:53:36.429186: Current learning rate: 0.00506 
+2024-09-09 02:00:36.452327: train_loss -0.7545 
+2024-09-09 02:00:36.466014: val_loss -0.7585 
+2024-09-09 02:00:36.466751: Pseudo dice [0.9637, 0.957, 0.9789, 0.949, 0.8861, 0.9681, 0.9898] 
+2024-09-09 02:00:36.467340: Epoch time: 420.03 s 
+2024-09-09 02:00:40.566417:  
+2024-09-09 02:00:40.595812: Epoch 532 
+2024-09-09 02:00:40.597094: Current learning rate: 0.00505 
+2024-09-09 02:07:42.646651: train_loss -0.7379 
+2024-09-09 02:07:42.658346: val_loss -0.6982 
+2024-09-09 02:07:42.659220: Pseudo dice [0.9651, 0.9617, 0.9755, 0.9513, 0.8796, 0.9652, 0.9921] 
+2024-09-09 02:07:42.660768: Epoch time: 422.12 s 
+2024-09-09 02:07:47.824237:  
+2024-09-09 02:07:47.832447: Epoch 533 
+2024-09-09 02:07:47.833329: Current learning rate: 0.00504 
+2024-09-09 02:15:26.030177: train_loss -0.7375 
+2024-09-09 02:15:26.049806: val_loss -0.7341 
+2024-09-09 02:15:26.052723: Pseudo dice [0.966, 0.957, 0.9788, 0.9501, 0.8792, 0.9731, 0.9923] 
+2024-09-09 02:15:26.056521: Epoch time: 458.21 s 
+2024-09-09 02:15:31.081652:  
+2024-09-09 02:15:31.093081: Epoch 534 
+2024-09-09 02:15:31.094097: Current learning rate: 0.00503 
+2024-09-09 02:22:16.154021: train_loss -0.742 
+2024-09-09 02:22:16.156181: val_loss -0.7342 
+2024-09-09 02:22:16.157982: Pseudo dice [0.9633, 0.9585, 0.9799, 0.9526, 0.8804, 0.9669, 0.9909] 
+2024-09-09 02:22:16.159049: Epoch time: 405.08 s 
+2024-09-09 02:22:20.971762:  
+2024-09-09 02:22:20.987857: Epoch 535 
+2024-09-09 02:22:20.989058: Current learning rate: 0.00502 
+2024-09-09 02:29:34.142971: train_loss -0.7405 
+2024-09-09 02:29:34.161362: val_loss -0.7314 
+2024-09-09 02:29:34.162304: Pseudo dice [0.9629, 0.9559, 0.9809, 0.9439, 0.8813, 0.9594, 0.9905] 
+2024-09-09 02:29:34.163161: Epoch time: 433.18 s 
+2024-09-09 02:29:38.012434:  
+2024-09-09 02:29:38.020862: Epoch 536 
+2024-09-09 02:29:38.033003: Current learning rate: 0.00501 
+2024-09-09 02:36:34.930632: train_loss -0.7403 
+2024-09-09 02:36:34.945403: val_loss -0.7599 
+2024-09-09 02:36:34.946938: Pseudo dice [0.9635, 0.9537, 0.9792, 0.9556, 0.8822, 0.9697, 0.9914] 
+2024-09-09 02:36:34.948141: Epoch time: 416.92 s 
+2024-09-09 02:36:40.128442:  
+2024-09-09 02:36:40.129683: Epoch 537 
+2024-09-09 02:36:40.130308: Current learning rate: 0.005 
+2024-09-09 02:43:43.711485: train_loss -0.7482 
+2024-09-09 02:43:43.731683: val_loss -0.7758 
+2024-09-09 02:43:43.732773: Pseudo dice [0.9628, 0.959, 0.9785, 0.9478, 0.8744, 0.9729, 0.9908] 
+2024-09-09 02:43:43.733778: Epoch time: 423.59 s 
+2024-09-09 02:43:47.941031:  
+2024-09-09 02:43:47.942713: Epoch 538 
+2024-09-09 02:43:47.943864: Current learning rate: 0.00499 
+2024-09-09 02:50:58.619169: train_loss -0.7519 
+2024-09-09 02:50:58.624243: val_loss -0.7546 
+2024-09-09 02:50:58.625406: Pseudo dice [0.9618, 0.9571, 0.9779, 0.9496, 0.8742, 0.9668, 0.9905] 
+2024-09-09 02:50:58.626124: Epoch time: 430.71 s 
+2024-09-09 02:51:02.172618:  
+2024-09-09 02:51:02.188685: Epoch 539 
+2024-09-09 02:51:02.189926: Current learning rate: 0.00498 
+2024-09-09 02:58:04.613096: train_loss -0.7457 
+2024-09-09 02:58:04.630820: val_loss -0.7363 
+2024-09-09 02:58:04.631771: Pseudo dice [0.9646, 0.9588, 0.9791, 0.9519, 0.8627, 0.9706, 0.9893] 
+2024-09-09 02:58:04.632935: Epoch time: 422.45 s 
+2024-09-09 02:58:08.603350:  
+2024-09-09 02:58:08.604928: Epoch 540 
+2024-09-09 02:58:08.605855: Current learning rate: 0.00497 
+2024-09-09 03:04:55.462516: train_loss -0.7337 
+2024-09-09 03:04:55.464782: val_loss -0.7251 
+2024-09-09 03:04:55.466579: Pseudo dice [0.9637, 0.959, 0.9769, 0.9515, 0.8862, 0.9727, 0.9922] 
+2024-09-09 03:04:55.468508: Epoch time: 406.9 s 
+2024-09-09 03:04:58.663852:  
+2024-09-09 03:04:58.665101: Epoch 541 
+2024-09-09 03:04:58.667375: Current learning rate: 0.00496 
+2024-09-09 03:11:54.142319: train_loss -0.7507 
+2024-09-09 03:11:54.162728: val_loss -0.7477 
+2024-09-09 03:11:54.163573: Pseudo dice [0.9634, 0.9574, 0.9775, 0.9491, 0.888, 0.9692, 0.9923] 
+2024-09-09 03:11:54.164125: Epoch time: 415.49 s 
+2024-09-09 03:11:54.164925: Yayy! New best EMA pseudo Dice: 0.955 
+2024-09-09 03:12:01.946622:  
+2024-09-09 03:12:01.947476: Epoch 542 
+2024-09-09 03:12:01.948216: Current learning rate: 0.00495 
+2024-09-09 03:18:54.112858: train_loss -0.7527 
+2024-09-09 03:18:54.125053: val_loss -0.7262 
+2024-09-09 03:18:54.125736: Pseudo dice [0.965, 0.96, 0.979, 0.9519, 0.8562, 0.9675, 0.9907] 
+2024-09-09 03:18:54.126278: Epoch time: 412.17 s 
+2024-09-09 03:18:59.753225:  
+2024-09-09 03:18:59.758618: Epoch 543 
+2024-09-09 03:18:59.759809: Current learning rate: 0.00494 
+2024-09-09 03:26:25.024886: train_loss -0.755 
+2024-09-09 03:26:25.041915: val_loss -0.7689 
+2024-09-09 03:26:25.042770: Pseudo dice [0.9669, 0.9586, 0.9806, 0.9465, 0.875, 0.9704, 0.9911] 
+2024-09-09 03:26:25.043872: Epoch time: 445.29 s 
+2024-09-09 03:26:29.238159:  
+2024-09-09 03:26:29.260485: Epoch 544 
+2024-09-09 03:26:29.261836: Current learning rate: 0.00493 
+2024-09-09 03:33:30.646930: train_loss -0.7501 
+2024-09-09 03:33:30.662839: val_loss -0.756 
+2024-09-09 03:33:30.664230: Pseudo dice [0.9634, 0.9576, 0.9785, 0.9547, 0.8706, 0.9682, 0.9902] 
+2024-09-09 03:33:30.665212: Epoch time: 421.45 s 
+2024-09-09 03:33:34.497860:  
+2024-09-09 03:33:34.499135: Epoch 545 
+2024-09-09 03:33:34.499970: Current learning rate: 0.00492 
+2024-09-09 03:40:44.608960: train_loss -0.7551 
+2024-09-09 03:40:44.627418: val_loss -0.7151 
+2024-09-09 03:40:44.629699: Pseudo dice [0.9634, 0.9572, 0.9772, 0.9511, 0.8639, 0.9691, 0.9907] 
+2024-09-09 03:40:44.631454: Epoch time: 430.12 s 
+2024-09-09 03:40:49.792561:  
+2024-09-09 03:40:49.808835: Epoch 546 
+2024-09-09 03:40:49.809810: Current learning rate: 0.00491 
+2024-09-09 03:48:07.343748: train_loss -0.7464 
+2024-09-09 03:48:07.352260: val_loss -0.7268 
+2024-09-09 03:48:07.358860: Pseudo dice [0.9634, 0.9585, 0.9781, 0.9454, 0.8661, 0.9691, 0.9884] 
+2024-09-09 03:48:07.360052: Epoch time: 437.53 s 
+2024-09-09 03:48:10.925629:  
+2024-09-09 03:48:10.941538: Epoch 547 
+2024-09-09 03:48:10.942811: Current learning rate: 0.0049 
+2024-09-09 03:55:23.184469: train_loss -0.7552 
+2024-09-09 03:55:23.193044: val_loss -0.7353 
+2024-09-09 03:55:23.194158: Pseudo dice [0.9622, 0.9575, 0.9764, 0.9444, 0.8695, 0.9741, 0.9896] 
+2024-09-09 03:55:23.195702: Epoch time: 432.26 s 
+2024-09-09 03:55:27.070409:  
+2024-09-09 03:55:27.084537: Epoch 548 
+2024-09-09 03:55:27.085498: Current learning rate: 0.00489 
+2024-09-09 04:02:45.789802: train_loss -0.7539 
+2024-09-09 04:02:45.795880: val_loss -0.726 
+2024-09-09 04:02:45.796606: Pseudo dice [0.9652, 0.9581, 0.9815, 0.9559, 0.8685, 0.9694, 0.9898] 
+2024-09-09 04:02:45.797302: Epoch time: 438.73 s 
+2024-09-09 04:02:48.858298:  
+2024-09-09 04:02:48.872604: Epoch 549 
+2024-09-09 04:02:48.874770: Current learning rate: 0.00488 
+2024-09-09 04:09:57.124061: train_loss -0.754 
+2024-09-09 04:09:57.145263: val_loss -0.7483 
+2024-09-09 04:09:57.146107: Pseudo dice [0.9621, 0.9595, 0.9754, 0.9485, 0.8588, 0.9685, 0.9893] 
+2024-09-09 04:09:57.147251: Epoch time: 428.27 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_9_06_02_53.txt

@@ -0,0 +1,375 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-09 06:02:56.631961: Using torch.compile... 
+2024-09-09 06:03:03.025392: do_dummy_2d_data_aug: False 
+2024-09-09 06:03:03.039730: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-09 06:03:03.044098: The split file contains 5 splits. 
+2024-09-09 06:03:03.044453: Desired fold for training: 0 
+2024-09-09 06:03:03.044780: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-09 06:03:25.906877: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-09 06:03:27.689810:  
+2024-09-09 06:03:27.694401: Epoch 550 
+2024-09-09 06:03:27.696295: Current learning rate: 0.00487 
+2024-09-09 06:20:17.913351: train_loss -0.7603 
+2024-09-09 06:20:17.914644: val_loss -0.7451 
+2024-09-09 06:20:17.915235: Pseudo dice [0.9663, 0.9605, 0.9814, 0.9543, 0.8816, 0.9704, 0.9906] 
+2024-09-09 06:20:17.915749: Epoch time: 1010.23 s 
+2024-09-09 06:20:19.751630:  
+2024-09-09 06:20:19.753316: Epoch 551 
+2024-09-09 06:20:19.754013: Current learning rate: 0.00486 
+2024-09-09 06:27:24.723177: train_loss -0.7402 
+2024-09-09 06:27:24.741417: val_loss -0.7546 
+2024-09-09 06:27:24.742229: Pseudo dice [0.9643, 0.9605, 0.9761, 0.9498, 0.8719, 0.9713, 0.9912] 
+2024-09-09 06:27:24.742959: Epoch time: 424.98 s 
+2024-09-09 06:27:28.735312:  
+2024-09-09 06:27:28.749013: Epoch 552 
+2024-09-09 06:27:28.751254: Current learning rate: 0.00485 
+2024-09-09 06:34:38.765999: train_loss -0.7352 
+2024-09-09 06:34:38.775469: val_loss -0.7566 
+2024-09-09 06:34:38.778468: Pseudo dice [0.9672, 0.9583, 0.9821, 0.95, 0.892, 0.974, 0.992] 
+2024-09-09 06:34:38.780082: Epoch time: 430.04 s 
+2024-09-09 06:34:38.781903: Yayy! New best EMA pseudo Dice: 0.9551 
+2024-09-09 06:34:48.572045:  
+2024-09-09 06:34:48.573036: Epoch 553 
+2024-09-09 06:34:48.573621: Current learning rate: 0.00484 
+2024-09-09 06:41:51.449909: train_loss -0.7506 
+2024-09-09 06:41:51.465674: val_loss -0.7387 
+2024-09-09 06:41:51.468258: Pseudo dice [0.9656, 0.9598, 0.9796, 0.9511, 0.8794, 0.9696, 0.99] 
+2024-09-09 06:41:51.473429: Epoch time: 422.88 s 
+2024-09-09 06:41:51.474679: Yayy! New best EMA pseudo Dice: 0.9552 
+2024-09-09 06:42:02.146034:  
+2024-09-09 06:42:02.146793: Epoch 554 
+2024-09-09 06:42:02.147368: Current learning rate: 0.00484 
+2024-09-09 06:49:16.500631: train_loss -0.7419 
+2024-09-09 06:49:16.505303: val_loss -0.7441 
+2024-09-09 06:49:16.509568: Pseudo dice [0.9632, 0.9578, 0.978, 0.9503, 0.8638, 0.9692, 0.9882] 
+2024-09-09 06:49:16.513167: Epoch time: 434.36 s 
+2024-09-09 06:49:20.232500:  
+2024-09-09 06:49:20.233488: Epoch 555 
+2024-09-09 06:49:20.234432: Current learning rate: 0.00483 
+2024-09-09 06:56:19.590824: train_loss -0.7409 
+2024-09-09 06:56:19.605192: val_loss -0.7666 
+2024-09-09 06:56:19.605913: Pseudo dice [0.9652, 0.9586, 0.9781, 0.9475, 0.8761, 0.9702, 0.9904] 
+2024-09-09 06:56:19.606752: Epoch time: 419.36 s 
+2024-09-09 06:56:23.776696:  
+2024-09-09 06:56:23.792892: Epoch 556 
+2024-09-09 06:56:23.795945: Current learning rate: 0.00482 
+2024-09-09 07:03:51.696842: train_loss -0.752 
+2024-09-09 07:03:51.713595: val_loss -0.7419 
+2024-09-09 07:03:51.716576: Pseudo dice [0.9622, 0.9549, 0.9791, 0.9524, 0.8759, 0.9702, 0.9895] 
+2024-09-09 07:03:51.718041: Epoch time: 447.96 s 
+2024-09-09 07:03:56.490799:  
+2024-09-09 07:03:56.492417: Epoch 557 
+2024-09-09 07:03:56.493801: Current learning rate: 0.00481 
+2024-09-09 07:11:19.434313: train_loss -0.7424 
+2024-09-09 07:11:19.437880: val_loss -0.7761 
+2024-09-09 07:11:19.439711: Pseudo dice [0.9667, 0.9586, 0.9825, 0.9497, 0.8835, 0.9716, 0.9912] 
+2024-09-09 07:11:19.441727: Epoch time: 442.95 s 
+2024-09-09 07:11:19.442419: Yayy! New best EMA pseudo Dice: 0.9553 
+2024-09-09 07:11:31.041974:  
+2024-09-09 07:11:31.044365: Epoch 558 
+2024-09-09 07:11:31.045725: Current learning rate: 0.0048 
+2024-09-09 07:18:20.227630: train_loss -0.7389 
+2024-09-09 07:18:20.250443: val_loss -0.7581 
+2024-09-09 07:18:20.251400: Pseudo dice [0.9611, 0.9563, 0.9759, 0.9471, 0.8744, 0.9691, 0.9906] 
+2024-09-09 07:18:20.252699: Epoch time: 409.19 s 
+2024-09-09 07:18:25.107147:  
+2024-09-09 07:18:25.109285: Epoch 559 
+2024-09-09 07:18:25.110723: Current learning rate: 0.00479 
+2024-09-09 07:25:18.813308: train_loss -0.7432 
+2024-09-09 07:25:18.842348: val_loss -0.7322 
+2024-09-09 07:25:18.844029: Pseudo dice [0.966, 0.9591, 0.9826, 0.9466, 0.8724, 0.9683, 0.9897] 
+2024-09-09 07:25:18.844870: Epoch time: 413.71 s 
+2024-09-09 07:25:23.513362:  
+2024-09-09 07:25:23.519943: Epoch 560 
+2024-09-09 07:25:23.529958: Current learning rate: 0.00478 
+2024-09-09 07:32:46.066507: train_loss -0.7347 
+2024-09-09 07:32:46.081511: val_loss -0.7397 
+2024-09-09 07:32:46.082279: Pseudo dice [0.9593, 0.9582, 0.9742, 0.9515, 0.8827, 0.9658, 0.9922] 
+2024-09-09 07:32:46.082835: Epoch time: 442.57 s 
+2024-09-09 07:32:49.756862:  
+2024-09-09 07:32:49.768819: Epoch 561 
+2024-09-09 07:32:49.769796: Current learning rate: 0.00477 
+2024-09-09 07:40:00.743493: train_loss -0.7223 
+2024-09-09 07:40:00.765564: val_loss -0.6993 
+2024-09-09 07:40:00.768086: Pseudo dice [0.9581, 0.9527, 0.9724, 0.9409, 0.8642, 0.9585, 0.9894] 
+2024-09-09 07:40:00.770141: Epoch time: 431.0 s 
+2024-09-09 07:40:03.840492:  
+2024-09-09 07:40:03.841902: Epoch 562 
+2024-09-09 07:40:03.842684: Current learning rate: 0.00476 
+2024-09-09 07:47:09.079212: train_loss -0.7206 
+2024-09-09 07:47:09.090026: val_loss -0.752 
+2024-09-09 07:47:09.090764: Pseudo dice [0.96, 0.9534, 0.9746, 0.9447, 0.8735, 0.9668, 0.9926] 
+2024-09-09 07:47:09.094072: Epoch time: 425.24 s 
+2024-09-09 07:47:13.835169:  
+2024-09-09 07:47:13.837013: Epoch 563 
+2024-09-09 07:47:13.838615: Current learning rate: 0.00475 
+2024-09-09 07:54:25.573789: train_loss -0.7255 
+2024-09-09 07:54:25.575709: val_loss -0.7457 
+2024-09-09 07:54:25.576385: Pseudo dice [0.9618, 0.9558, 0.9766, 0.9446, 0.8649, 0.9616, 0.9899] 
+2024-09-09 07:54:25.577358: Epoch time: 431.75 s 
+2024-09-09 07:54:29.343794:  
+2024-09-09 07:54:29.360898: Epoch 564 
+2024-09-09 07:54:29.361995: Current learning rate: 0.00474 
+2024-09-09 08:01:48.338226: train_loss -0.7342 
+2024-09-09 08:01:48.341051: val_loss -0.7061 
+2024-09-09 08:01:48.342759: Pseudo dice [0.9648, 0.9561, 0.9802, 0.9486, 0.8752, 0.9561, 0.9915] 
+2024-09-09 08:01:48.344805: Epoch time: 439.05 s 
+2024-09-09 08:01:53.479789:  
+2024-09-09 08:01:53.496809: Epoch 565 
+2024-09-09 08:01:53.498163: Current learning rate: 0.00473 
+2024-09-09 08:09:05.839331: train_loss -0.7511 
+2024-09-09 08:09:05.857362: val_loss -0.699 
+2024-09-09 08:09:05.869035: Pseudo dice [0.964, 0.9597, 0.977, 0.9514, 0.8756, 0.9708, 0.9915] 
+2024-09-09 08:09:05.869846: Epoch time: 432.37 s 
+2024-09-09 08:09:15.274359:  
+2024-09-09 08:09:15.276090: Epoch 566 
+2024-09-09 08:09:15.276901: Current learning rate: 0.00472 
+2024-09-09 08:16:09.583379: train_loss -0.7315 
+2024-09-09 08:16:09.609302: val_loss -0.7208 
+2024-09-09 08:16:09.611184: Pseudo dice [0.9615, 0.9541, 0.9741, 0.9443, 0.8665, 0.9691, 0.9907] 
+2024-09-09 08:16:09.612337: Epoch time: 414.31 s 
+2024-09-09 08:16:14.462742:  
+2024-09-09 08:16:14.477462: Epoch 567 
+2024-09-09 08:16:14.479534: Current learning rate: 0.00471 
+2024-09-09 08:23:16.848541: train_loss -0.7439 
+2024-09-09 08:23:16.865277: val_loss -0.7794 
+2024-09-09 08:23:16.868563: Pseudo dice [0.962, 0.9578, 0.9753, 0.9481, 0.8651, 0.9599, 0.9897] 
+2024-09-09 08:23:16.871205: Epoch time: 422.39 s 
+2024-09-09 08:23:21.493345:  
+2024-09-09 08:23:21.494296: Epoch 568 
+2024-09-09 08:23:21.494945: Current learning rate: 0.0047 
+2024-09-09 08:30:40.391170: train_loss -0.7447 
+2024-09-09 08:30:40.410253: val_loss -0.7391 
+2024-09-09 08:30:40.412991: Pseudo dice [0.964, 0.9587, 0.9803, 0.9488, 0.8684, 0.9552, 0.9918] 
+2024-09-09 08:30:40.418377: Epoch time: 438.9 s 
+2024-09-09 08:30:43.999080:  
+2024-09-09 08:30:44.012684: Epoch 569 
+2024-09-09 08:30:44.013535: Current learning rate: 0.00469 
+2024-09-09 08:37:37.768157: train_loss -0.7203 
+2024-09-09 08:37:37.769874: val_loss -0.7374 
+2024-09-09 08:37:37.770739: Pseudo dice [0.9654, 0.9581, 0.9807, 0.9553, 0.8806, 0.9714, 0.9898] 
+2024-09-09 08:37:37.771320: Epoch time: 413.78 s 
+2024-09-09 08:37:40.483245:  
+2024-09-09 08:37:40.484608: Epoch 570 
+2024-09-09 08:37:40.485559: Current learning rate: 0.00468 
+2024-09-09 08:44:37.175408: train_loss -0.7268 
+2024-09-09 08:44:37.193683: val_loss -0.724 
+2024-09-09 08:44:37.194942: Pseudo dice [0.9616, 0.9545, 0.9779, 0.9466, 0.8654, 0.9686, 0.9886] 
+2024-09-09 08:44:37.196084: Epoch time: 416.7 s 
+2024-09-09 08:44:41.940787:  
+2024-09-09 08:44:41.957372: Epoch 571 
+2024-09-09 08:44:41.958112: Current learning rate: 0.00467 
+2024-09-09 08:51:22.110503: train_loss -0.7372 
+2024-09-09 08:51:22.121001: val_loss -0.7325 
+2024-09-09 08:51:22.121934: Pseudo dice [0.9625, 0.9567, 0.9792, 0.9413, 0.8767, 0.9676, 0.9907] 
+2024-09-09 08:51:22.123304: Epoch time: 400.17 s 
+2024-09-09 08:51:25.947127:  
+2024-09-09 08:51:25.971808: Epoch 572 
+2024-09-09 08:51:25.973206: Current learning rate: 0.00466 
+2024-09-09 08:58:48.936965: train_loss -0.7385 
+2024-09-09 08:58:48.955417: val_loss -0.7664 
+2024-09-09 08:58:48.956340: Pseudo dice [0.9633, 0.9565, 0.9798, 0.9491, 0.8695, 0.9697, 0.9886] 
+2024-09-09 08:58:48.957205: Epoch time: 442.99 s 
+2024-09-09 08:58:52.549312:  
+2024-09-09 08:58:52.560695: Epoch 573 
+2024-09-09 08:58:52.561872: Current learning rate: 0.00465 
+2024-09-09 09:05:59.966615: train_loss -0.7402 
+2024-09-09 09:05:59.969907: val_loss -0.7162 
+2024-09-09 09:05:59.971672: Pseudo dice [0.9638, 0.9579, 0.9793, 0.9487, 0.8753, 0.9697, 0.9908] 
+2024-09-09 09:05:59.973351: Epoch time: 427.42 s 
+2024-09-09 09:06:04.258865:  
+2024-09-09 09:06:04.272478: Epoch 574 
+2024-09-09 09:06:04.273300: Current learning rate: 0.00464 
+2024-09-09 09:13:07.119146: train_loss -0.7349 
+2024-09-09 09:13:07.125919: val_loss -0.7689 
+2024-09-09 09:13:07.128433: Pseudo dice [0.962, 0.956, 0.9778, 0.9532, 0.8782, 0.9673, 0.9906] 
+2024-09-09 09:13:07.130149: Epoch time: 422.87 s 
+2024-09-09 09:13:11.680481:  
+2024-09-09 09:13:11.692973: Epoch 575 
+2024-09-09 09:13:11.694064: Current learning rate: 0.00463 
+2024-09-09 09:19:59.884737: train_loss -0.7394 
+2024-09-09 09:19:59.913286: val_loss -0.7055 
+2024-09-09 09:19:59.914445: Pseudo dice [0.9591, 0.9582, 0.9731, 0.9487, 0.8721, 0.9675, 0.9891] 
+2024-09-09 09:19:59.915174: Epoch time: 408.21 s 
+2024-09-09 09:20:05.039048:  
+2024-09-09 09:20:05.043177: Epoch 576 
+2024-09-09 09:20:05.044502: Current learning rate: 0.00462 
+2024-09-09 09:27:29.440382: train_loss -0.7141 
+2024-09-09 09:27:29.445983: val_loss -0.7719 
+2024-09-09 09:27:29.447628: Pseudo dice [0.9643, 0.9592, 0.9761, 0.9478, 0.8556, 0.9678, 0.9882] 
+2024-09-09 09:27:29.448671: Epoch time: 444.48 s 
+2024-09-09 09:27:34.413881:  
+2024-09-09 09:27:34.415181: Epoch 577 
+2024-09-09 09:27:34.416935: Current learning rate: 0.00461 
+2024-09-09 09:34:43.934671: train_loss -0.7463 
+2024-09-09 09:34:43.940225: val_loss -0.7685 
+2024-09-09 09:34:43.941898: Pseudo dice [0.9637, 0.9531, 0.9798, 0.9484, 0.8725, 0.9681, 0.9918] 
+2024-09-09 09:34:43.942795: Epoch time: 429.53 s 
+2024-09-09 09:34:49.086019:  
+2024-09-09 09:34:49.096748: Epoch 578 
+2024-09-09 09:34:49.097638: Current learning rate: 0.0046 
+2024-09-09 09:41:45.468446: train_loss -0.7412 
+2024-09-09 09:41:45.489094: val_loss -0.7146 
+2024-09-09 09:41:45.490330: Pseudo dice [0.9638, 0.9595, 0.9745, 0.9518, 0.862, 0.9659, 0.9897] 
+2024-09-09 09:41:45.493163: Epoch time: 416.39 s 
+2024-09-09 09:41:49.429968:  
+2024-09-09 09:41:49.431081: Epoch 579 
+2024-09-09 09:41:49.431961: Current learning rate: 0.00459 
+2024-09-09 09:48:39.335858: train_loss -0.7389 
+2024-09-09 09:48:39.353025: val_loss -0.7392 
+2024-09-09 09:48:39.355876: Pseudo dice [0.9621, 0.9564, 0.9766, 0.9471, 0.8639, 0.9683, 0.9906] 
+2024-09-09 09:48:39.358107: Epoch time: 409.91 s 
+2024-09-09 09:48:46.567490:  
+2024-09-09 09:48:46.568933: Epoch 580 
+2024-09-09 09:48:46.569597: Current learning rate: 0.00458 
+2024-09-09 09:55:58.945194: train_loss -0.724 
+2024-09-09 09:55:58.961113: val_loss -0.745 
+2024-09-09 09:55:58.962118: Pseudo dice [0.9631, 0.9581, 0.9779, 0.9544, 0.8763, 0.968, 0.9905] 
+2024-09-09 09:55:58.963211: Epoch time: 432.38 s 
+2024-09-09 09:56:04.345268:  
+2024-09-09 09:56:04.361156: Epoch 581 
+2024-09-09 09:56:04.361786: Current learning rate: 0.00457 
+2024-09-09 10:03:14.467088: train_loss -0.7464 
+2024-09-09 10:03:14.481934: val_loss -0.7254 
+2024-09-09 10:03:14.482704: Pseudo dice [0.9662, 0.9577, 0.9769, 0.9492, 0.8768, 0.9721, 0.9911] 
+2024-09-09 10:03:14.483270: Epoch time: 430.13 s 
+2024-09-09 10:03:19.818254:  
+2024-09-09 10:03:19.821606: Epoch 582 
+2024-09-09 10:03:19.822964: Current learning rate: 0.00456 
+2024-09-09 10:11:33.882074: train_loss -0.7563 
+2024-09-09 10:11:33.888083: val_loss -0.742 
+2024-09-09 10:11:33.889486: Pseudo dice [0.9648, 0.9589, 0.979, 0.95, 0.8739, 0.9697, 0.9881] 
+2024-09-09 10:11:33.890876: Epoch time: 494.07 s 
+2024-09-09 10:11:40.523521:  
+2024-09-09 10:11:40.541254: Epoch 583 
+2024-09-09 10:11:40.542951: Current learning rate: 0.00455 
+2024-09-09 10:20:03.945407: train_loss -0.7403 
+2024-09-09 10:20:03.963030: val_loss -0.7278 
+2024-09-09 10:20:03.964042: Pseudo dice [0.9642, 0.9588, 0.9784, 0.9444, 0.8742, 0.968, 0.9915] 
+2024-09-09 10:20:03.964858: Epoch time: 503.43 s 
+2024-09-09 10:20:08.677163:  
+2024-09-09 10:20:08.679908: Epoch 584 
+2024-09-09 10:20:08.682932: Current learning rate: 0.00454 
+2024-09-09 10:28:11.210809: train_loss -0.7444 
+2024-09-09 10:28:11.219982: val_loss -0.7712 
+2024-09-09 10:28:11.221347: Pseudo dice [0.9652, 0.956, 0.9782, 0.9492, 0.8879, 0.9731, 0.9919] 
+2024-09-09 10:28:11.223473: Epoch time: 482.54 s 
+2024-09-09 10:28:16.790022:  
+2024-09-09 10:28:16.806266: Epoch 585 
+2024-09-09 10:28:16.807581: Current learning rate: 0.00453 
+2024-09-09 10:36:15.782964: train_loss -0.7488 
+2024-09-09 10:36:15.798307: val_loss -0.7148 
+2024-09-09 10:36:15.799568: Pseudo dice [0.9661, 0.9605, 0.982, 0.9529, 0.8761, 0.9728, 0.9905] 
+2024-09-09 10:36:15.801071: Epoch time: 479.0 s 
+2024-09-09 10:36:19.288614:  
+2024-09-09 10:36:19.290022: Epoch 586 
+2024-09-09 10:36:19.290700: Current learning rate: 0.00452 
+2024-09-09 10:43:53.465867: train_loss -0.7423 
+2024-09-09 10:43:53.477727: val_loss -0.7528 
+2024-09-09 10:43:53.479944: Pseudo dice [0.9628, 0.9549, 0.9777, 0.9431, 0.8751, 0.972, 0.9919] 
+2024-09-09 10:43:53.480514: Epoch time: 454.19 s 
+2024-09-09 10:43:58.706894:  
+2024-09-09 10:43:58.721505: Epoch 587 
+2024-09-09 10:43:58.723094: Current learning rate: 0.00451 
+2024-09-09 10:51:28.002802: train_loss -0.7496 
+2024-09-09 10:51:28.004390: val_loss -0.7496 
+2024-09-09 10:51:28.005829: Pseudo dice [0.9637, 0.9582, 0.979, 0.9519, 0.8757, 0.9701, 0.9899] 
+2024-09-09 10:51:28.007830: Epoch time: 449.31 s 
+2024-09-09 10:51:33.282417:  
+2024-09-09 10:51:33.284086: Epoch 588 
+2024-09-09 10:51:33.285663: Current learning rate: 0.0045 
+2024-09-09 10:59:20.195071: train_loss -0.745 
+2024-09-09 10:59:20.196671: val_loss -0.7597 
+2024-09-09 10:59:20.197358: Pseudo dice [0.9642, 0.9579, 0.9791, 0.9545, 0.8799, 0.9686, 0.9903] 
+2024-09-09 10:59:20.197896: Epoch time: 466.92 s 
+2024-09-09 10:59:26.500173:  
+2024-09-09 10:59:26.531780: Epoch 589 
+2024-09-09 10:59:26.532927: Current learning rate: 0.00449 
+2024-09-09 11:07:25.035481: train_loss -0.7518 
+2024-09-09 11:07:25.061036: val_loss -0.7698 
+2024-09-09 11:07:25.063335: Pseudo dice [0.9615, 0.9559, 0.9779, 0.9451, 0.8671, 0.967, 0.9912] 
+2024-09-09 11:07:25.065454: Epoch time: 478.56 s 
+2024-09-09 11:07:31.057538:  
+2024-09-09 11:07:31.061849: Epoch 590 
+2024-09-09 11:07:31.064251: Current learning rate: 0.00448 
+2024-09-09 11:15:48.503275: train_loss -0.728 
+2024-09-09 11:15:48.525023: val_loss -0.7601 
+2024-09-09 11:15:48.526494: Pseudo dice [0.966, 0.9589, 0.9802, 0.9534, 0.8615, 0.9694, 0.9916] 
+2024-09-09 11:15:48.528924: Epoch time: 497.45 s 
+2024-09-09 11:15:54.384403:  
+2024-09-09 11:15:54.385720: Epoch 591 
+2024-09-09 11:15:54.387324: Current learning rate: 0.00447 
+2024-09-09 11:24:45.620971: train_loss -0.7458 
+2024-09-09 11:24:45.648223: val_loss -0.7691 
+2024-09-09 11:24:45.650250: Pseudo dice [0.9627, 0.9587, 0.9793, 0.948, 0.8801, 0.9687, 0.9904] 
+2024-09-09 11:24:45.651764: Epoch time: 531.24 s 
+2024-09-09 11:24:51.709139:  
+2024-09-09 11:24:51.711725: Epoch 592 
+2024-09-09 11:24:51.713498: Current learning rate: 0.00446 
+2024-09-09 11:34:07.159482: train_loss -0.7466 
+2024-09-09 11:34:07.177322: val_loss -0.75 
+2024-09-09 11:34:07.183793: Pseudo dice [0.9651, 0.9585, 0.9784, 0.9492, 0.8789, 0.9704, 0.9924] 
+2024-09-09 11:34:07.186668: Epoch time: 555.49 s 
+2024-09-09 11:34:14.259421:  
+2024-09-09 11:34:14.261096: Epoch 593 
+2024-09-09 11:34:14.262521: Current learning rate: 0.00445 
+2024-09-09 11:43:46.516201: train_loss -0.7277 
+2024-09-09 11:43:46.520841: val_loss -0.7676 
+2024-09-09 11:43:46.522246: Pseudo dice [0.9628, 0.9553, 0.979, 0.9457, 0.8839, 0.9718, 0.9915] 
+2024-09-09 11:43:46.524162: Epoch time: 572.26 s 
+2024-09-09 11:43:55.221014:  
+2024-09-09 11:43:55.222780: Epoch 594 
+2024-09-09 11:43:55.223958: Current learning rate: 0.00444 
+2024-09-09 11:53:15.024175: train_loss -0.7491 
+2024-09-09 11:53:15.028970: val_loss -0.7463 
+2024-09-09 11:53:15.031193: Pseudo dice [0.9663, 0.9584, 0.9782, 0.9553, 0.8763, 0.9651, 0.9911] 
+2024-09-09 11:53:15.034906: Epoch time: 559.81 s 
+2024-09-09 11:53:20.195601:  
+2024-09-09 11:53:20.198505: Epoch 595 
+2024-09-09 11:53:20.199568: Current learning rate: 0.00443 
+2024-09-09 12:02:29.224554: train_loss -0.7449 
+2024-09-09 12:02:29.408757: val_loss -0.7367 
+2024-09-09 12:02:29.416837: Pseudo dice [0.9621, 0.9577, 0.9758, 0.9492, 0.8813, 0.9667, 0.9902] 
+2024-09-09 12:02:29.448704: Epoch time: 549.05 s 
+2024-09-09 12:02:37.644966:  
+2024-09-09 12:02:37.646846: Epoch 596 
+2024-09-09 12:02:37.650980: Current learning rate: 0.00442 
+2024-09-09 12:12:15.481374: train_loss -0.7303 
+2024-09-09 12:12:15.494081: val_loss -0.7459 
+2024-09-09 12:12:15.494882: Pseudo dice [0.9693, 0.9624, 0.9831, 0.9555, 0.8758, 0.9664, 0.9906] 
+2024-09-09 12:12:15.495456: Epoch time: 577.92 s 
+2024-09-09 12:12:19.206968:  
+2024-09-09 12:12:19.220928: Epoch 597 
+2024-09-09 12:12:19.221877: Current learning rate: 0.00441 
+2024-09-09 12:21:44.688620: train_loss -0.7411 
+2024-09-09 12:21:44.721741: val_loss -0.7606 
+2024-09-09 12:21:44.724080: Pseudo dice [0.9638, 0.958, 0.9796, 0.9491, 0.8696, 0.9707, 0.9899] 
+2024-09-09 12:21:44.726192: Epoch time: 565.49 s 
+2024-09-09 12:21:51.303209:  
+2024-09-09 12:21:51.317165: Epoch 598 
+2024-09-09 12:21:51.318915: Current learning rate: 0.0044 
+2024-09-09 12:31:25.272320: train_loss -0.7325 
+2024-09-09 12:31:25.274565: val_loss -0.767 
+2024-09-09 12:31:25.276626: Pseudo dice [0.9618, 0.9538, 0.977, 0.9497, 0.8864, 0.9706, 0.9921] 
+2024-09-09 12:31:25.316452: Epoch time: 574.01 s 
+2024-09-09 12:31:30.110684:  
+2024-09-09 12:31:30.128879: Epoch 599 
+2024-09-09 12:31:30.132127: Current learning rate: 0.00439 
+2024-09-09 12:40:50.062476: train_loss -0.7468 
+2024-09-09 12:40:50.064575: val_loss -0.739 
+2024-09-09 12:40:50.066683: Pseudo dice [0.9629, 0.9572, 0.9784, 0.9533, 0.8841, 0.9732, 0.9924] 
+2024-09-09 12:40:50.068943: Epoch time: 560.02 s 
+2024-09-09 12:40:56.341649: Yayy! New best EMA pseudo Dice: 0.9554 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_9_14_03_19.txt

@@ -0,0 +1,379 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-09 14:03:22.076601: Using torch.compile... 
+2024-09-09 14:03:28.212011: do_dummy_2d_data_aug: False 
+2024-09-09 14:03:28.217779: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-09 14:03:28.219218: The split file contains 5 splits. 
+2024-09-09 14:03:28.219802: Desired fold for training: 0 
+2024-09-09 14:03:28.220347: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-09 14:03:38.612104: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-09 14:03:39.715178:  
+2024-09-09 14:03:39.727012: Epoch 600 
+2024-09-09 14:03:39.728021: Current learning rate: 0.00438 
+2024-09-09 14:11:59.416926: train_loss -0.7422 
+2024-09-09 14:11:59.419363: val_loss -0.7325 
+2024-09-09 14:11:59.420147: Pseudo dice [0.9657, 0.9593, 0.9795, 0.9518, 0.8836, 0.9712, 0.9904] 
+2024-09-09 14:11:59.420814: Epoch time: 499.71 s 
+2024-09-09 14:11:59.421353: Yayy! New best EMA pseudo Dice: 0.9556 
+2024-09-09 14:12:02.452893:  
+2024-09-09 14:12:02.453717: Epoch 601 
+2024-09-09 14:12:02.454346: Current learning rate: 0.00437 
+2024-09-09 14:16:35.120170: train_loss -0.7364 
+2024-09-09 14:16:35.134027: val_loss -0.7579 
+2024-09-09 14:16:35.134976: Pseudo dice [0.9672, 0.9607, 0.9823, 0.9545, 0.8774, 0.9684, 0.9889] 
+2024-09-09 14:16:35.135902: Epoch time: 272.67 s 
+2024-09-09 14:16:35.137845: Yayy! New best EMA pseudo Dice: 0.9557 
+2024-09-09 14:16:40.408855:  
+2024-09-09 14:16:40.424344: Epoch 602 
+2024-09-09 14:16:40.425403: Current learning rate: 0.00436 
+2024-09-09 14:21:45.694649: train_loss -0.748 
+2024-09-09 14:21:45.696903: val_loss -0.7883 
+2024-09-09 14:21:45.697672: Pseudo dice [0.9649, 0.9589, 0.9786, 0.9435, 0.8691, 0.9687, 0.9905] 
+2024-09-09 14:21:45.698351: Epoch time: 305.29 s 
+2024-09-09 14:21:48.685863:  
+2024-09-09 14:21:48.687200: Epoch 603 
+2024-09-09 14:21:48.688124: Current learning rate: 0.00435 
+2024-09-09 14:26:52.402236: train_loss -0.7314 
+2024-09-09 14:26:52.409422: val_loss -0.7335 
+2024-09-09 14:26:52.410270: Pseudo dice [0.965, 0.9585, 0.9798, 0.9526, 0.8799, 0.97, 0.9912] 
+2024-09-09 14:26:52.410928: Epoch time: 303.72 s 
+2024-09-09 14:26:55.218347:  
+2024-09-09 14:26:55.219481: Epoch 604 
+2024-09-09 14:26:55.220280: Current learning rate: 0.00434 
+2024-09-09 14:31:48.754349: train_loss -0.7133 
+2024-09-09 14:31:48.756225: val_loss -0.7526 
+2024-09-09 14:31:48.756888: Pseudo dice [0.9592, 0.9506, 0.9745, 0.9426, 0.848, 0.9635, 0.9862] 
+2024-09-09 14:31:48.757500: Epoch time: 293.54 s 
+2024-09-09 14:31:54.272987:  
+2024-09-09 14:31:54.274343: Epoch 605 
+2024-09-09 14:31:54.275200: Current learning rate: 0.00433 
+2024-09-09 14:36:48.683113: train_loss -0.7182 
+2024-09-09 14:36:48.685116: val_loss -0.7704 
+2024-09-09 14:36:48.685783: Pseudo dice [0.9578, 0.9519, 0.9732, 0.9411, 0.8512, 0.9573, 0.9881] 
+2024-09-09 14:36:48.686482: Epoch time: 294.42 s 
+2024-09-09 14:36:51.799385:  
+2024-09-09 14:36:51.833958: Epoch 606 
+2024-09-09 14:36:51.835211: Current learning rate: 0.00432 
+2024-09-09 14:41:30.004753: train_loss -0.7218 
+2024-09-09 14:41:30.029559: val_loss -0.7377 
+2024-09-09 14:41:30.030435: Pseudo dice [0.9591, 0.9539, 0.9741, 0.9439, 0.8711, 0.9659, 0.99] 
+2024-09-09 14:41:30.031131: Epoch time: 278.21 s 
+2024-09-09 14:41:33.277447:  
+2024-09-09 14:41:33.295761: Epoch 607 
+2024-09-09 14:41:33.296906: Current learning rate: 0.00431 
+2024-09-09 14:46:29.152950: train_loss -0.734 
+2024-09-09 14:46:29.167913: val_loss -0.7718 
+2024-09-09 14:46:29.169107: Pseudo dice [0.9621, 0.9563, 0.9758, 0.9456, 0.8737, 0.9702, 0.9899] 
+2024-09-09 14:46:29.169804: Epoch time: 295.88 s 
+2024-09-09 14:46:32.245308:  
+2024-09-09 14:46:32.246365: Epoch 608 
+2024-09-09 14:46:32.247142: Current learning rate: 0.0043 
+2024-09-09 14:51:39.533293: train_loss -0.7411 
+2024-09-09 14:51:39.535317: val_loss -0.7571 
+2024-09-09 14:51:39.536027: Pseudo dice [0.9651, 0.9572, 0.9794, 0.9465, 0.8695, 0.9696, 0.9902] 
+2024-09-09 14:51:39.536741: Epoch time: 307.29 s 
+2024-09-09 14:51:42.682621:  
+2024-09-09 14:51:42.684038: Epoch 609 
+2024-09-09 14:51:42.684955: Current learning rate: 0.00429 
+2024-09-09 14:56:31.931863: train_loss -0.7451 
+2024-09-09 14:56:31.934075: val_loss -0.7675 
+2024-09-09 14:56:31.935049: Pseudo dice [0.9628, 0.9583, 0.9752, 0.9461, 0.8762, 0.9727, 0.9918] 
+2024-09-09 14:56:31.935938: Epoch time: 289.25 s 
+2024-09-09 14:56:35.297440:  
+2024-09-09 14:56:35.298479: Epoch 610 
+2024-09-09 14:56:35.299164: Current learning rate: 0.00429 
+2024-09-09 15:01:21.293403: train_loss -0.7548 
+2024-09-09 15:01:21.309429: val_loss -0.7716 
+2024-09-09 15:01:21.310223: Pseudo dice [0.9661, 0.9597, 0.9798, 0.9474, 0.8825, 0.9701, 0.9895] 
+2024-09-09 15:01:21.310869: Epoch time: 286.0 s 
+2024-09-09 15:01:25.081353:  
+2024-09-09 15:01:25.091021: Epoch 611 
+2024-09-09 15:01:25.092171: Current learning rate: 0.00428 
+2024-09-09 15:05:58.622586: train_loss -0.7429 
+2024-09-09 15:05:58.645321: val_loss -0.7182 
+2024-09-09 15:05:58.646045: Pseudo dice [0.9635, 0.9589, 0.975, 0.9479, 0.8663, 0.9682, 0.9903] 
+2024-09-09 15:05:58.646835: Epoch time: 273.55 s 
+2024-09-09 15:06:02.454710:  
+2024-09-09 15:06:02.455966: Epoch 612 
+2024-09-09 15:06:02.456789: Current learning rate: 0.00427 
+2024-09-09 15:10:37.636665: train_loss -0.7381 
+2024-09-09 15:10:37.646020: val_loss -0.7652 
+2024-09-09 15:10:37.646905: Pseudo dice [0.9647, 0.9583, 0.9796, 0.9522, 0.8871, 0.9715, 0.9911] 
+2024-09-09 15:10:37.648996: Epoch time: 275.19 s 
+2024-09-09 15:10:41.478444:  
+2024-09-09 15:10:41.492481: Epoch 613 
+2024-09-09 15:10:41.493296: Current learning rate: 0.00426 
+2024-09-09 15:14:58.072337: train_loss -0.7366 
+2024-09-09 15:14:58.085584: val_loss -0.7364 
+2024-09-09 15:14:58.086416: Pseudo dice [0.9623, 0.9585, 0.9767, 0.9464, 0.8753, 0.9703, 0.991] 
+2024-09-09 15:14:58.087026: Epoch time: 256.6 s 
+2024-09-09 15:15:00.576244:  
+2024-09-09 15:15:00.577509: Epoch 614 
+2024-09-09 15:15:00.578336: Current learning rate: 0.00425 
+2024-09-09 15:19:23.149049: train_loss -0.7539 
+2024-09-09 15:19:23.150834: val_loss -0.7586 
+2024-09-09 15:19:23.151572: Pseudo dice [0.9637, 0.9574, 0.9783, 0.9462, 0.8747, 0.9726, 0.9916] 
+2024-09-09 15:19:23.152222: Epoch time: 262.58 s 
+2024-09-09 15:19:25.674244:  
+2024-09-09 15:19:25.675217: Epoch 615 
+2024-09-09 15:19:25.675809: Current learning rate: 0.00424 
+2024-09-09 15:23:55.370193: train_loss -0.7586 
+2024-09-09 15:23:55.381642: val_loss -0.7567 
+2024-09-09 15:23:55.392031: Pseudo dice [0.9628, 0.9584, 0.9738, 0.9504, 0.8854, 0.9735, 0.9902] 
+2024-09-09 15:23:55.393099: Epoch time: 269.7 s 
+2024-09-09 15:23:58.334517:  
+2024-09-09 15:23:58.355759: Epoch 616 
+2024-09-09 15:23:58.356756: Current learning rate: 0.00423 
+2024-09-09 15:28:55.204910: train_loss -0.747 
+2024-09-09 15:28:55.207375: val_loss -0.7437 
+2024-09-09 15:28:55.208094: Pseudo dice [0.964, 0.96, 0.9793, 0.9513, 0.8802, 0.968, 0.9897] 
+2024-09-09 15:28:55.208695: Epoch time: 296.88 s 
+2024-09-09 15:28:58.940400:  
+2024-09-09 15:28:58.941791: Epoch 617 
+2024-09-09 15:28:58.942800: Current learning rate: 0.00422 
+2024-09-09 15:34:06.003500: train_loss -0.7403 
+2024-09-09 15:34:06.005576: val_loss -0.7604 
+2024-09-09 15:34:06.006370: Pseudo dice [0.9605, 0.9584, 0.9721, 0.9439, 0.8702, 0.9632, 0.9902] 
+2024-09-09 15:34:06.007088: Epoch time: 307.08 s 
+2024-09-09 15:34:08.603883:  
+2024-09-09 15:34:08.605194: Epoch 618 
+2024-09-09 15:34:08.606259: Current learning rate: 0.00421 
+2024-09-09 15:39:05.335115: train_loss -0.7271 
+2024-09-09 15:39:05.337793: val_loss -0.7294 
+2024-09-09 15:39:05.339176: Pseudo dice [0.9592, 0.9571, 0.9747, 0.945, 0.8709, 0.9699, 0.9907] 
+2024-09-09 15:39:05.340258: Epoch time: 296.74 s 
+2024-09-09 15:39:09.705041:  
+2024-09-09 15:39:09.706099: Epoch 619 
+2024-09-09 15:39:09.706744: Current learning rate: 0.0042 
+2024-09-09 15:44:23.080055: train_loss -0.7331 
+2024-09-09 15:44:23.097711: val_loss -0.7584 
+2024-09-09 15:44:23.098778: Pseudo dice [0.9623, 0.9562, 0.9753, 0.9457, 0.8652, 0.9651, 0.9895] 
+2024-09-09 15:44:23.100104: Epoch time: 313.38 s 
+2024-09-09 15:44:26.440796:  
+2024-09-09 15:44:26.452566: Epoch 620 
+2024-09-09 15:44:26.453738: Current learning rate: 0.00419 
+2024-09-09 15:49:10.806569: train_loss -0.7382 
+2024-09-09 15:49:10.809245: val_loss -0.7542 
+2024-09-09 15:49:10.810108: Pseudo dice [0.9638, 0.9553, 0.9793, 0.9503, 0.8776, 0.9726, 0.9914] 
+2024-09-09 15:49:10.810765: Epoch time: 284.37 s 
+2024-09-09 15:49:14.124080:  
+2024-09-09 15:49:14.125385: Epoch 621 
+2024-09-09 15:49:14.126131: Current learning rate: 0.00418 
+2024-09-09 15:54:29.241578: train_loss -0.7476 
+2024-09-09 15:54:29.257683: val_loss -0.7506 
+2024-09-09 15:54:29.258484: Pseudo dice [0.965, 0.957, 0.9787, 0.9516, 0.8769, 0.9706, 0.9916] 
+2024-09-09 15:54:29.259104: Epoch time: 315.13 s 
+2024-09-09 15:54:32.508845:  
+2024-09-09 15:54:32.510001: Epoch 622 
+2024-09-09 15:54:32.510721: Current learning rate: 0.00417 
+2024-09-09 15:59:31.662357: train_loss -0.7481 
+2024-09-09 15:59:31.694438: val_loss -0.7549 
+2024-09-09 15:59:31.695517: Pseudo dice [0.9668, 0.9607, 0.9803, 0.9529, 0.8761, 0.9698, 0.9902] 
+2024-09-09 15:59:31.696429: Epoch time: 299.16 s 
+2024-09-09 15:59:35.837550:  
+2024-09-09 15:59:35.838664: Epoch 623 
+2024-09-09 15:59:35.839518: Current learning rate: 0.00416 
+2024-09-09 16:04:20.507539: train_loss -0.7357 
+2024-09-09 16:04:20.524469: val_loss -0.7438 
+2024-09-09 16:04:20.525537: Pseudo dice [0.9608, 0.9555, 0.9748, 0.9406, 0.8637, 0.9672, 0.9864] 
+2024-09-09 16:04:20.526342: Epoch time: 284.68 s 
+2024-09-09 16:04:23.241707:  
+2024-09-09 16:04:23.242855: Epoch 624 
+2024-09-09 16:04:23.243686: Current learning rate: 0.00415 
+2024-09-09 16:09:36.133538: train_loss -0.7336 
+2024-09-09 16:09:36.149508: val_loss -0.6819 
+2024-09-09 16:09:36.150252: Pseudo dice [0.9638, 0.9582, 0.9782, 0.9484, 0.8817, 0.9725, 0.992] 
+2024-09-09 16:09:36.150864: Epoch time: 312.9 s 
+2024-09-09 16:09:39.595434:  
+2024-09-09 16:09:39.620378: Epoch 625 
+2024-09-09 16:09:39.637002: Current learning rate: 0.00414 
+2024-09-09 16:14:24.890019: train_loss -0.7329 
+2024-09-09 16:14:24.902401: val_loss -0.7844 
+2024-09-09 16:14:24.903314: Pseudo dice [0.9637, 0.957, 0.9782, 0.9508, 0.8879, 0.9708, 0.9926] 
+2024-09-09 16:14:24.904173: Epoch time: 285.3 s 
+2024-09-09 16:14:29.681528:  
+2024-09-09 16:14:29.682505: Epoch 626 
+2024-09-09 16:14:29.683182: Current learning rate: 0.00413 
+2024-09-09 16:19:38.026428: train_loss -0.7441 
+2024-09-09 16:19:38.058457: val_loss -0.7094 
+2024-09-09 16:19:38.059409: Pseudo dice [0.9633, 0.9582, 0.9766, 0.9501, 0.8785, 0.9713, 0.9914] 
+2024-09-09 16:19:38.060169: Epoch time: 308.35 s 
+2024-09-09 16:19:41.150708:  
+2024-09-09 16:19:41.159856: Epoch 627 
+2024-09-09 16:19:41.160829: Current learning rate: 0.00412 
+2024-09-09 16:24:43.345859: train_loss -0.7552 
+2024-09-09 16:24:43.347795: val_loss -0.7791 
+2024-09-09 16:24:43.348482: Pseudo dice [0.9687, 0.9616, 0.9827, 0.95, 0.8834, 0.9703, 0.9908] 
+2024-09-09 16:24:43.349091: Epoch time: 302.2 s 
+2024-09-09 16:24:46.359220:  
+2024-09-09 16:24:46.360339: Epoch 628 
+2024-09-09 16:24:46.361092: Current learning rate: 0.00411 
+2024-09-09 16:29:55.546157: train_loss -0.7562 
+2024-09-09 16:29:55.549491: val_loss -0.6949 
+2024-09-09 16:29:55.550644: Pseudo dice [0.9651, 0.9578, 0.9789, 0.9463, 0.8586, 0.972, 0.9884] 
+2024-09-09 16:29:55.551581: Epoch time: 309.2 s 
+2024-09-09 16:29:58.661152:  
+2024-09-09 16:29:58.662293: Epoch 629 
+2024-09-09 16:29:58.663255: Current learning rate: 0.0041 
+2024-09-09 16:35:50.020266: train_loss -0.7384 
+2024-09-09 16:35:50.055580: val_loss -0.7043 
+2024-09-09 16:35:50.056670: Pseudo dice [0.9649, 0.9592, 0.9796, 0.9493, 0.8761, 0.9715, 0.9896] 
+2024-09-09 16:35:50.057805: Epoch time: 351.36 s 
+2024-09-09 16:35:53.288487:  
+2024-09-09 16:35:53.299706: Epoch 630 
+2024-09-09 16:35:53.300564: Current learning rate: 0.00409 
+2024-09-09 16:42:47.649271: train_loss -0.7594 
+2024-09-09 16:42:47.653062: val_loss -0.7455 
+2024-09-09 16:42:47.653816: Pseudo dice [0.9654, 0.9582, 0.9799, 0.9498, 0.8737, 0.9708, 0.9907] 
+2024-09-09 16:42:47.654556: Epoch time: 414.37 s 
+2024-09-09 16:42:53.078510:  
+2024-09-09 16:42:53.081245: Epoch 631 
+2024-09-09 16:42:53.082629: Current learning rate: 0.00408 
+2024-09-09 16:50:26.585078: train_loss -0.7426 
+2024-09-09 16:50:26.587542: val_loss -0.7652 
+2024-09-09 16:50:26.588605: Pseudo dice [0.9653, 0.9591, 0.9795, 0.9573, 0.8799, 0.9707, 0.9916] 
+2024-09-09 16:50:26.589410: Epoch time: 453.53 s 
+2024-09-09 16:50:38.345997:  
+2024-09-09 16:50:38.348694: Epoch 632 
+2024-09-09 16:50:38.349601: Current learning rate: 0.00407 
+2024-09-09 16:58:54.092952: train_loss -0.7271 
+2024-09-09 16:58:54.097241: val_loss -0.7649 
+2024-09-09 16:58:54.098630: Pseudo dice [0.9639, 0.9589, 0.9759, 0.9506, 0.8875, 0.9736, 0.992] 
+2024-09-09 16:58:54.099923: Epoch time: 495.76 s 
+2024-09-09 16:59:00.257211:  
+2024-09-09 16:59:00.259009: Epoch 633 
+2024-09-09 16:59:00.260136: Current learning rate: 0.00406 
+2024-09-09 17:07:27.570461: train_loss -0.7437 
+2024-09-09 17:07:27.578601: val_loss -0.7651 
+2024-09-09 17:07:27.582284: Pseudo dice [0.965, 0.9566, 0.9782, 0.9481, 0.8689, 0.9691, 0.9897] 
+2024-09-09 17:07:27.586010: Epoch time: 507.32 s 
+2024-09-09 17:07:33.934267:  
+2024-09-09 17:07:33.935801: Epoch 634 
+2024-09-09 17:07:33.936939: Current learning rate: 0.00405 
+2024-09-09 17:15:35.128105: train_loss -0.7422 
+2024-09-09 17:15:35.132767: val_loss -0.7507 
+2024-09-09 17:15:35.134817: Pseudo dice [0.963, 0.957, 0.9783, 0.9575, 0.8766, 0.9725, 0.9911] 
+2024-09-09 17:15:35.135737: Epoch time: 481.2 s 
+2024-09-09 17:15:40.973423:  
+2024-09-09 17:15:40.974965: Epoch 635 
+2024-09-09 17:15:40.977189: Current learning rate: 0.00404 
+2024-09-09 17:24:06.149046: train_loss -0.7481 
+2024-09-09 17:24:06.151471: val_loss -0.7689 
+2024-09-09 17:24:06.153125: Pseudo dice [0.9648, 0.9573, 0.9804, 0.9515, 0.8827, 0.9705, 0.9908] 
+2024-09-09 17:24:06.154147: Epoch time: 505.18 s 
+2024-09-09 17:24:12.615417:  
+2024-09-09 17:24:12.617611: Epoch 636 
+2024-09-09 17:24:12.618599: Current learning rate: 0.00403 
+2024-09-09 17:32:36.255636: train_loss -0.7364 
+2024-09-09 17:32:36.260107: val_loss -0.7631 
+2024-09-09 17:32:36.261040: Pseudo dice [0.9641, 0.9593, 0.9796, 0.9527, 0.8781, 0.9697, 0.9915] 
+2024-09-09 17:32:36.261916: Epoch time: 503.66 s 
+2024-09-09 17:32:39.180224:  
+2024-09-09 17:32:39.181864: Epoch 637 
+2024-09-09 17:32:39.183678: Current learning rate: 0.00402 
+2024-09-09 17:38:06.837975: train_loss -0.7451 
+2024-09-09 17:38:06.842221: val_loss -0.7501 
+2024-09-09 17:38:06.843815: Pseudo dice [0.9621, 0.9582, 0.9759, 0.9511, 0.8863, 0.9732, 0.9917] 
+2024-09-09 17:38:06.845093: Epoch time: 327.67 s 
+2024-09-09 17:38:06.847601: Yayy! New best EMA pseudo Dice: 0.9558 
+2024-09-09 17:38:13.609845:  
+2024-09-09 17:38:13.610870: Epoch 638 
+2024-09-09 17:38:13.612443: Current learning rate: 0.00401 
+2024-09-09 17:43:32.459765: train_loss -0.7533 
+2024-09-09 17:43:32.462000: val_loss -0.7757 
+2024-09-09 17:43:32.462749: Pseudo dice [0.9656, 0.958, 0.9805, 0.9562, 0.8802, 0.9734, 0.989] 
+2024-09-09 17:43:32.463422: Epoch time: 318.86 s 
+2024-09-09 17:43:32.464028: Yayy! New best EMA pseudo Dice: 0.9559 
+2024-09-09 17:43:39.542725:  
+2024-09-09 17:43:39.544365: Epoch 639 
+2024-09-09 17:43:39.545213: Current learning rate: 0.004 
+2024-09-09 17:48:51.193573: train_loss -0.7541 
+2024-09-09 17:48:51.213397: val_loss -0.7693 
+2024-09-09 17:48:51.215893: Pseudo dice [0.964, 0.9591, 0.978, 0.9498, 0.8759, 0.9703, 0.9907] 
+2024-09-09 17:48:51.217686: Epoch time: 311.66 s 
+2024-09-09 17:48:55.016494:  
+2024-09-09 17:48:55.018605: Epoch 640 
+2024-09-09 17:48:55.020026: Current learning rate: 0.00399 
+2024-09-09 17:54:08.133231: train_loss -0.7377 
+2024-09-09 17:54:08.135722: val_loss -0.7561 
+2024-09-09 17:54:08.136395: Pseudo dice [0.9652, 0.959, 0.9794, 0.9503, 0.8755, 0.9703, 0.9907] 
+2024-09-09 17:54:08.137085: Epoch time: 313.16 s 
+2024-09-09 17:54:11.002209:  
+2024-09-09 17:54:11.003322: Epoch 641 
+2024-09-09 17:54:11.004114: Current learning rate: 0.00398 
+2024-09-09 17:59:33.455332: train_loss -0.7475 
+2024-09-09 17:59:33.458094: val_loss -0.7531 
+2024-09-09 17:59:33.458885: Pseudo dice [0.965, 0.9572, 0.9791, 0.9523, 0.8883, 0.9676, 0.9928] 
+2024-09-09 17:59:33.459574: Epoch time: 322.46 s 
+2024-09-09 17:59:33.460237: Yayy! New best EMA pseudo Dice: 0.956 
+2024-09-09 17:59:39.763316:  
+2024-09-09 17:59:39.764744: Epoch 642 
+2024-09-09 17:59:39.765556: Current learning rate: 0.00397 
+2024-09-09 18:04:57.942556: train_loss -0.7359 
+2024-09-09 18:04:57.945580: val_loss -0.7542 
+2024-09-09 18:04:57.946388: Pseudo dice [0.9663, 0.9591, 0.9792, 0.9513, 0.8897, 0.9712, 0.9918] 
+2024-09-09 18:04:57.947139: Epoch time: 318.18 s 
+2024-09-09 18:04:57.947829: Yayy! New best EMA pseudo Dice: 0.9563 
+2024-09-09 18:05:04.315767:  
+2024-09-09 18:05:04.317210: Epoch 643 
+2024-09-09 18:05:04.318023: Current learning rate: 0.00396 
+2024-09-09 18:10:20.711021: train_loss -0.7477 
+2024-09-09 18:10:20.712394: val_loss -0.7503 
+2024-09-09 18:10:20.713129: Pseudo dice [0.9659, 0.959, 0.9795, 0.9447, 0.8793, 0.9734, 0.9917] 
+2024-09-09 18:10:20.713948: Epoch time: 316.4 s 
+2024-09-09 18:10:22.817966:  
+2024-09-09 18:10:22.819468: Epoch 644 
+2024-09-09 18:10:22.820858: Current learning rate: 0.00395 
+2024-09-09 18:15:39.370100: train_loss -0.746 
+2024-09-09 18:15:39.373694: val_loss -0.7478 
+2024-09-09 18:15:39.375593: Pseudo dice [0.9655, 0.9583, 0.9805, 0.9392, 0.8769, 0.9612, 0.9895] 
+2024-09-09 18:15:39.377716: Epoch time: 316.56 s 
+2024-09-09 18:15:44.098774:  
+2024-09-09 18:15:44.101436: Epoch 645 
+2024-09-09 18:15:44.102561: Current learning rate: 0.00394 
+2024-09-09 18:21:02.724511: train_loss -0.736 
+2024-09-09 18:21:02.727781: val_loss -0.7321 
+2024-09-09 18:21:02.728988: Pseudo dice [0.9646, 0.9577, 0.9787, 0.9504, 0.8722, 0.9698, 0.99] 
+2024-09-09 18:21:02.729839: Epoch time: 318.64 s 
+2024-09-09 18:21:05.474806:  
+2024-09-09 18:21:05.476289: Epoch 646 
+2024-09-09 18:21:05.477223: Current learning rate: 0.00393 
+2024-09-09 18:26:31.738006: train_loss -0.7574 
+2024-09-09 18:26:31.739866: val_loss -0.7487 
+2024-09-09 18:26:31.740853: Pseudo dice [0.9675, 0.9592, 0.9803, 0.9554, 0.8775, 0.9739, 0.9896] 
+2024-09-09 18:26:31.741754: Epoch time: 326.27 s 
+2024-09-09 18:26:34.422112:  
+2024-09-09 18:26:34.423337: Epoch 647 
+2024-09-09 18:26:34.424088: Current learning rate: 0.00392 
+2024-09-09 18:31:52.812871: train_loss -0.7459 
+2024-09-09 18:31:52.815935: val_loss -0.7325 
+2024-09-09 18:31:52.816846: Pseudo dice [0.9609, 0.9578, 0.9759, 0.9438, 0.8974, 0.9726, 0.9927] 
+2024-09-09 18:31:52.817971: Epoch time: 318.4 s 
+2024-09-09 18:31:55.960718:  
+2024-09-09 18:31:55.962753: Epoch 648 
+2024-09-09 18:31:55.963823: Current learning rate: 0.00391 
+2024-09-09 18:37:07.309243: train_loss -0.7547 
+2024-09-09 18:37:07.315973: val_loss -0.7708 
+2024-09-09 18:37:07.317032: Pseudo dice [0.9659, 0.9591, 0.9795, 0.956, 0.8844, 0.9751, 0.9917] 
+2024-09-09 18:37:07.318601: Epoch time: 311.36 s 
+2024-09-09 18:37:07.319734: Yayy! New best EMA pseudo Dice: 0.9564 
+2024-09-09 18:37:13.013046:  
+2024-09-09 18:37:13.015067: Epoch 649 
+2024-09-09 18:37:13.016473: Current learning rate: 0.0039 
+2024-09-09 18:42:27.664673: train_loss -0.76 
+2024-09-09 18:42:27.667174: val_loss -0.7477 
+2024-09-09 18:42:27.668156: Pseudo dice [0.9687, 0.961, 0.982, 0.9582, 0.8891, 0.9761, 0.9928] 
+2024-09-09 18:42:27.669147: Epoch time: 314.67 s 
+2024-09-09 18:42:31.091205: Yayy! New best EMA pseudo Dice: 0.9569 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_0/logs/training_log_2024_9_9_22_02_26.txt

@@ -0,0 +1,378 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-09 22:02:30.504366: Using torch.compile... 
+2024-09-09 22:02:38.536471: do_dummy_2d_data_aug: False 
+2024-09-09 22:02:38.553031: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-09 22:02:38.563509: The split file contains 5 splits. 
+2024-09-09 22:02:38.564045: Desired fold for training: 0 
+2024-09-09 22:02:38.564492: This split has 1324 training and 332 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-09 22:02:53.915228: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-09 22:02:54.779288:  
+2024-09-09 22:02:54.780607: Epoch 650 
+2024-09-09 22:02:54.781664: Current learning rate: 0.00389 
+2024-09-09 22:14:22.540025: train_loss -0.7455 
+2024-09-09 22:14:22.542140: val_loss -0.7692 
+2024-09-09 22:14:22.542728: Pseudo dice [0.9651, 0.9592, 0.9801, 0.941, 0.8846, 0.975, 0.9926] 
+2024-09-09 22:14:22.543295: Epoch time: 687.79 s 
+2024-09-09 22:14:22.543883: Yayy! New best EMA pseudo Dice: 0.9569 
+2024-09-09 22:14:27.730831:  
+2024-09-09 22:14:27.731484: Epoch 651 
+2024-09-09 22:14:27.731899: Current learning rate: 0.00388 
+2024-09-09 22:19:36.442305: train_loss -0.7479 
+2024-09-09 22:19:36.444684: val_loss -0.751 
+2024-09-09 22:19:36.446591: Pseudo dice [0.963, 0.9586, 0.9775, 0.9543, 0.8758, 0.969, 0.989] 
+2024-09-09 22:19:36.447936: Epoch time: 308.71 s 
+2024-09-09 22:19:39.256821:  
+2024-09-09 22:19:39.258682: Epoch 652 
+2024-09-09 22:19:39.259560: Current learning rate: 0.00387 
+2024-09-09 22:25:08.892207: train_loss -0.7526 
+2024-09-09 22:25:08.895167: val_loss -0.7471 
+2024-09-09 22:25:08.896349: Pseudo dice [0.9636, 0.9592, 0.977, 0.9521, 0.8792, 0.9637, 0.9877] 
+2024-09-09 22:25:08.897243: Epoch time: 329.65 s 
+2024-09-09 22:25:11.231498:  
+2024-09-09 22:25:11.232445: Epoch 653 
+2024-09-09 22:25:11.233183: Current learning rate: 0.00386 
+2024-09-09 22:30:18.995815: train_loss -0.7504 
+2024-09-09 22:30:18.999334: val_loss -0.7329 
+2024-09-09 22:30:19.000483: Pseudo dice [0.9649, 0.9602, 0.9779, 0.953, 0.8871, 0.9724, 0.9921] 
+2024-09-09 22:30:19.001879: Epoch time: 307.77 s 
+2024-09-09 22:30:21.590392:  
+2024-09-09 22:30:21.591694: Epoch 654 
+2024-09-09 22:30:21.593402: Current learning rate: 0.00385 
+2024-09-09 22:35:27.492182: train_loss -0.7507 
+2024-09-09 22:35:27.494580: val_loss -0.7604 
+2024-09-09 22:35:27.495290: Pseudo dice [0.9668, 0.9611, 0.9795, 0.9518, 0.8893, 0.9747, 0.9923] 
+2024-09-09 22:35:27.497130: Epoch time: 305.91 s 
+2024-09-09 22:35:27.497811: Yayy! New best EMA pseudo Dice: 0.9569 
+2024-09-09 22:35:32.491446:  
+2024-09-09 22:35:32.492679: Epoch 655 
+2024-09-09 22:35:32.493257: Current learning rate: 0.00384 
+2024-09-09 22:40:47.639734: train_loss -0.7509 
+2024-09-09 22:40:47.642272: val_loss -0.7269 
+2024-09-09 22:40:47.652800: Pseudo dice [0.9645, 0.9573, 0.9799, 0.9506, 0.8732, 0.969, 0.9902] 
+2024-09-09 22:40:47.653910: Epoch time: 315.15 s 
+2024-09-09 22:40:49.670734:  
+2024-09-09 22:40:49.671553: Epoch 656 
+2024-09-09 22:40:49.672069: Current learning rate: 0.00383 
+2024-09-09 22:46:01.609303: train_loss -0.7526 
+2024-09-09 22:46:01.611939: val_loss -0.7683 
+2024-09-09 22:46:01.613046: Pseudo dice [0.964, 0.9572, 0.9792, 0.9509, 0.8848, 0.9678, 0.9926] 
+2024-09-09 22:46:01.614046: Epoch time: 311.94 s 
+2024-09-09 22:46:04.051718:  
+2024-09-09 22:46:04.053025: Epoch 657 
+2024-09-09 22:46:04.054495: Current learning rate: 0.00382 
+2024-09-09 22:51:23.494652: train_loss -0.7458 
+2024-09-09 22:51:23.496440: val_loss -0.743 
+2024-09-09 22:51:23.496903: Pseudo dice [0.9657, 0.9591, 0.9796, 0.9474, 0.876, 0.9659, 0.9861] 
+2024-09-09 22:51:23.497379: Epoch time: 319.45 s 
+2024-09-09 22:51:25.933519:  
+2024-09-09 22:51:25.934756: Epoch 658 
+2024-09-09 22:51:25.935389: Current learning rate: 0.00381 
+2024-09-09 22:56:18.098736: train_loss -0.7574 
+2024-09-09 22:56:18.117017: val_loss -0.7375 
+2024-09-09 22:56:18.118707: Pseudo dice [0.9683, 0.9617, 0.9814, 0.9527, 0.8852, 0.9718, 0.9923] 
+2024-09-09 22:56:18.119505: Epoch time: 292.17 s 
+2024-09-09 22:56:22.491721:  
+2024-09-09 22:56:22.492935: Epoch 659 
+2024-09-09 22:56:22.493730: Current learning rate: 0.0038 
+2024-09-09 23:01:30.419495: train_loss -0.74 
+2024-09-09 23:01:30.421592: val_loss -0.7434 
+2024-09-09 23:01:30.422400: Pseudo dice [0.9667, 0.9591, 0.9805, 0.9499, 0.8883, 0.9726, 0.9924] 
+2024-09-09 23:01:30.423352: Epoch time: 307.93 s 
+2024-09-09 23:01:33.426079:  
+2024-09-09 23:01:33.427255: Epoch 660 
+2024-09-09 23:01:33.427850: Current learning rate: 0.00379 
+2024-09-09 23:06:26.770225: train_loss -0.7508 
+2024-09-09 23:06:26.773126: val_loss -0.7397 
+2024-09-09 23:06:26.774027: Pseudo dice [0.9657, 0.9601, 0.9791, 0.9445, 0.8712, 0.971, 0.9918] 
+2024-09-09 23:06:26.775289: Epoch time: 293.35 s 
+2024-09-09 23:06:29.821057:  
+2024-09-09 23:06:29.822011: Epoch 661 
+2024-09-09 23:06:29.822683: Current learning rate: 0.00378 
+2024-09-09 23:11:39.608925: train_loss -0.7396 
+2024-09-09 23:11:39.612111: val_loss -0.7642 
+2024-09-09 23:11:39.613157: Pseudo dice [0.963, 0.9594, 0.9753, 0.9548, 0.8864, 0.9718, 0.9917] 
+2024-09-09 23:11:39.614154: Epoch time: 309.79 s 
+2024-09-09 23:11:44.600822:  
+2024-09-09 23:11:44.602450: Epoch 662 
+2024-09-09 23:11:44.603137: Current learning rate: 0.00377 
+2024-09-09 23:16:44.431016: train_loss -0.7558 
+2024-09-09 23:16:44.432510: val_loss -0.7106 
+2024-09-09 23:16:44.433169: Pseudo dice [0.9633, 0.9582, 0.9765, 0.9478, 0.874, 0.9634, 0.9894] 
+2024-09-09 23:16:44.434615: Epoch time: 299.84 s 
+2024-09-09 23:16:46.009891:  
+2024-09-09 23:16:46.010668: Epoch 663 
+2024-09-09 23:16:46.011672: Current learning rate: 0.00376 
+2024-09-09 23:22:08.423873: train_loss -0.7433 
+2024-09-09 23:22:08.425832: val_loss -0.7791 
+2024-09-09 23:22:08.427141: Pseudo dice [0.9685, 0.961, 0.9823, 0.9482, 0.8704, 0.9715, 0.993] 
+2024-09-09 23:22:08.427829: Epoch time: 322.42 s 
+2024-09-09 23:22:10.939173:  
+2024-09-09 23:22:10.940185: Epoch 664 
+2024-09-09 23:22:10.940918: Current learning rate: 0.00375 
+2024-09-09 23:27:18.711446: train_loss -0.7496 
+2024-09-09 23:27:18.713885: val_loss -0.7651 
+2024-09-09 23:27:18.714545: Pseudo dice [0.9683, 0.9606, 0.982, 0.955, 0.8886, 0.9731, 0.9931] 
+2024-09-09 23:27:18.715077: Epoch time: 307.78 s 
+2024-09-09 23:27:21.015511:  
+2024-09-09 23:27:21.016696: Epoch 665 
+2024-09-09 23:27:21.017322: Current learning rate: 0.00374 
+2024-09-09 23:32:20.055047: train_loss -0.7358 
+2024-09-09 23:32:20.060367: val_loss -0.7681 
+2024-09-09 23:32:20.061120: Pseudo dice [0.9665, 0.9588, 0.9801, 0.9523, 0.8793, 0.9703, 0.9917] 
+2024-09-09 23:32:20.062119: Epoch time: 299.05 s 
+2024-09-09 23:32:21.752975:  
+2024-09-09 23:32:21.753976: Epoch 666 
+2024-09-09 23:32:21.754600: Current learning rate: 0.00373 
+2024-09-09 23:37:33.970813: train_loss -0.7439 
+2024-09-09 23:37:33.972721: val_loss -0.7782 
+2024-09-09 23:37:33.973320: Pseudo dice [0.9663, 0.9603, 0.9791, 0.9519, 0.8749, 0.9735, 0.9909] 
+2024-09-09 23:37:33.973810: Epoch time: 312.22 s 
+2024-09-09 23:37:35.568632:  
+2024-09-09 23:37:35.569721: Epoch 667 
+2024-09-09 23:37:35.570318: Current learning rate: 0.00372 
+2024-09-09 23:42:39.844890: train_loss -0.7502 
+2024-09-09 23:42:39.845814: val_loss -0.7296 
+2024-09-09 23:42:39.846316: Pseudo dice [0.9657, 0.9588, 0.9777, 0.9528, 0.8503, 0.9705, 0.9869] 
+2024-09-09 23:42:39.846986: Epoch time: 304.28 s 
+2024-09-09 23:42:41.645841:  
+2024-09-09 23:42:41.646742: Epoch 668 
+2024-09-09 23:42:41.647248: Current learning rate: 0.00371 
+2024-09-09 23:47:33.076339: train_loss -0.7532 
+2024-09-09 23:47:33.079078: val_loss -0.7407 
+2024-09-09 23:47:33.079861: Pseudo dice [0.9686, 0.9575, 0.9828, 0.9543, 0.8818, 0.9734, 0.991] 
+2024-09-09 23:47:33.080653: Epoch time: 291.43 s 
+2024-09-09 23:47:35.267970:  
+2024-09-09 23:47:35.269120: Epoch 669 
+2024-09-09 23:47:35.269863: Current learning rate: 0.0037 
+2024-09-09 23:52:57.021216: train_loss -0.742 
+2024-09-09 23:52:57.022505: val_loss -0.7496 
+2024-09-09 23:52:57.023003: Pseudo dice [0.9658, 0.9588, 0.9814, 0.9558, 0.8782, 0.9711, 0.9913] 
+2024-09-09 23:52:57.023520: Epoch time: 321.76 s 
+2024-09-09 23:52:59.497163:  
+2024-09-09 23:52:59.499933: Epoch 670 
+2024-09-09 23:52:59.501264: Current learning rate: 0.00369 
+2024-09-09 23:58:05.950350: train_loss -0.746 
+2024-09-09 23:58:05.952013: val_loss -0.7614 
+2024-09-09 23:58:05.952921: Pseudo dice [0.9671, 0.9583, 0.9817, 0.9501, 0.8872, 0.973, 0.9911] 
+2024-09-09 23:58:05.954615: Epoch time: 306.5 s 
+2024-09-09 23:58:09.064539:  
+2024-09-09 23:58:09.066676: Epoch 671 
+2024-09-09 23:58:09.067545: Current learning rate: 0.00368 
+2024-09-10 00:03:11.638621: train_loss -0.7451 
+2024-09-10 00:03:11.655130: val_loss -0.7315 
+2024-09-10 00:03:11.668555: Pseudo dice [0.964, 0.9572, 0.9804, 0.954, 0.8674, 0.9693, 0.9883] 
+2024-09-10 00:03:11.669424: Epoch time: 302.58 s 
+2024-09-10 00:03:14.126904:  
+2024-09-10 00:03:14.128334: Epoch 672 
+2024-09-10 00:03:14.129396: Current learning rate: 0.00367 
+2024-09-10 00:08:35.397063: train_loss -0.7412 
+2024-09-10 00:08:35.399824: val_loss -0.7492 
+2024-09-10 00:08:35.400788: Pseudo dice [0.9651, 0.9585, 0.9783, 0.9534, 0.8806, 0.9728, 0.9903] 
+2024-09-10 00:08:35.401423: Epoch time: 321.28 s 
+2024-09-10 00:08:37.740830:  
+2024-09-10 00:08:37.741859: Epoch 673 
+2024-09-10 00:08:37.742382: Current learning rate: 0.00366 
+2024-09-10 00:13:30.040856: train_loss -0.7602 
+2024-09-10 00:13:30.045089: val_loss -0.7511 
+2024-09-10 00:13:30.046138: Pseudo dice [0.9651, 0.9598, 0.977, 0.9497, 0.8852, 0.9677, 0.9924] 
+2024-09-10 00:13:30.046991: Epoch time: 292.3 s 
+2024-09-10 00:13:31.948530:  
+2024-09-10 00:13:31.950986: Epoch 674 
+2024-09-10 00:13:31.951591: Current learning rate: 0.00365 
+2024-09-10 00:18:49.059054: train_loss -0.7298 
+2024-09-10 00:18:49.061367: val_loss -0.7731 
+2024-09-10 00:18:49.061970: Pseudo dice [0.9661, 0.9601, 0.9795, 0.9485, 0.8678, 0.9699, 0.9904] 
+2024-09-10 00:18:49.062526: Epoch time: 317.12 s 
+2024-09-10 00:18:50.665370:  
+2024-09-10 00:18:50.667296: Epoch 675 
+2024-09-10 00:18:50.667847: Current learning rate: 0.00364 
+2024-09-10 00:24:01.334194: train_loss -0.7406 
+2024-09-10 00:24:01.336218: val_loss -0.7519 
+2024-09-10 00:24:01.336763: Pseudo dice [0.9631, 0.9605, 0.9764, 0.9561, 0.8834, 0.9696, 0.9895] 
+2024-09-10 00:24:01.337320: Epoch time: 310.67 s 
+2024-09-10 00:24:06.574935:  
+2024-09-10 00:24:06.576070: Epoch 676 
+2024-09-10 00:24:06.576634: Current learning rate: 0.00363 
+2024-09-10 00:29:46.384372: train_loss -0.7608 
+2024-09-10 00:29:46.401513: val_loss -0.7518 
+2024-09-10 00:29:46.402094: Pseudo dice [0.9662, 0.9617, 0.9815, 0.9531, 0.8881, 0.9734, 0.9918] 
+2024-09-10 00:29:46.402793: Epoch time: 339.81 s 
+2024-09-10 00:29:48.955803:  
+2024-09-10 00:29:48.957118: Epoch 677 
+2024-09-10 00:29:48.957787: Current learning rate: 0.00362 
+2024-09-10 00:34:12.094804: train_loss -0.7426 
+2024-09-10 00:34:12.102292: val_loss -0.7242 
+2024-09-10 00:34:12.102987: Pseudo dice [0.9636, 0.9585, 0.9785, 0.945, 0.8886, 0.9721, 0.991] 
+2024-09-10 00:34:12.103492: Epoch time: 263.14 s 
+2024-09-10 00:34:14.495895:  
+2024-09-10 00:34:14.496659: Epoch 678 
+2024-09-10 00:34:14.497171: Current learning rate: 0.00361 
+2024-09-10 00:38:56.906136: train_loss -0.7512 
+2024-09-10 00:38:56.908399: val_loss -0.7459 
+2024-09-10 00:38:56.909414: Pseudo dice [0.9636, 0.9623, 0.979, 0.9512, 0.8763, 0.9709, 0.9904] 
+2024-09-10 00:38:56.909931: Epoch time: 282.42 s 
+2024-09-10 00:39:01.139002:  
+2024-09-10 00:39:01.140134: Epoch 679 
+2024-09-10 00:39:01.140792: Current learning rate: 0.0036 
+2024-09-10 00:44:05.565263: train_loss -0.7479 
+2024-09-10 00:44:05.584229: val_loss -0.7328 
+2024-09-10 00:44:05.585788: Pseudo dice [0.9651, 0.9574, 0.9806, 0.9461, 0.8809, 0.9694, 0.9913] 
+2024-09-10 00:44:05.587556: Epoch time: 304.43 s 
+2024-09-10 00:44:09.684393:  
+2024-09-10 00:44:09.685212: Epoch 680 
+2024-09-10 00:44:09.685719: Current learning rate: 0.00359 
+2024-09-10 00:48:57.732038: train_loss -0.7558 
+2024-09-10 00:48:57.734887: val_loss -0.7846 
+2024-09-10 00:48:57.735481: Pseudo dice [0.9623, 0.9589, 0.9769, 0.9547, 0.8811, 0.9709, 0.9909] 
+2024-09-10 00:48:57.736181: Epoch time: 288.07 s 
+2024-09-10 00:49:00.236826:  
+2024-09-10 00:49:00.254577: Epoch 681 
+2024-09-10 00:49:00.255731: Current learning rate: 0.00358 
+2024-09-10 00:54:07.638384: train_loss -0.7554 
+2024-09-10 00:54:07.639819: val_loss -0.7848 
+2024-09-10 00:54:07.640355: Pseudo dice [0.9647, 0.96, 0.9801, 0.9538, 0.8888, 0.973, 0.9917] 
+2024-09-10 00:54:07.641067: Epoch time: 307.43 s 
+2024-09-10 00:54:10.672413:  
+2024-09-10 00:54:10.673163: Epoch 682 
+2024-09-10 00:54:10.673633: Current learning rate: 0.00357 
+2024-09-10 00:59:05.362923: train_loss -0.7534 
+2024-09-10 00:59:05.365310: val_loss -0.7625 
+2024-09-10 00:59:05.366199: Pseudo dice [0.9662, 0.9607, 0.9803, 0.951, 0.8763, 0.9694, 0.9901] 
+2024-09-10 00:59:05.366746: Epoch time: 294.69 s 
+2024-09-10 00:59:08.501666:  
+2024-09-10 00:59:08.503011: Epoch 683 
+2024-09-10 00:59:08.503871: Current learning rate: 0.00356 
+2024-09-10 01:04:12.570729: train_loss -0.7579 
+2024-09-10 01:04:12.572309: val_loss -0.7805 
+2024-09-10 01:04:12.572880: Pseudo dice [0.9652, 0.959, 0.9801, 0.9566, 0.8813, 0.9698, 0.9904] 
+2024-09-10 01:04:12.573430: Epoch time: 304.08 s 
+2024-09-10 01:04:15.292505:  
+2024-09-10 01:04:15.293412: Epoch 684 
+2024-09-10 01:04:15.293960: Current learning rate: 0.00355 
+2024-09-10 01:09:15.219102: train_loss -0.7617 
+2024-09-10 01:09:15.233573: val_loss -0.7476 
+2024-09-10 01:09:15.234190: Pseudo dice [0.9661, 0.9591, 0.9802, 0.9494, 0.8917, 0.9748, 0.9924] 
+2024-09-10 01:09:15.234674: Epoch time: 299.93 s 
+2024-09-10 01:09:15.235125: Yayy! New best EMA pseudo Dice: 0.9571 
+2024-09-10 01:09:21.513952:  
+2024-09-10 01:09:21.514624: Epoch 685 
+2024-09-10 01:09:21.515165: Current learning rate: 0.00354 
+2024-09-10 01:15:01.913410: train_loss -0.7464 
+2024-09-10 01:15:01.915752: val_loss -0.7654 
+2024-09-10 01:15:01.916366: Pseudo dice [0.9632, 0.9576, 0.978, 0.9433, 0.8806, 0.9746, 0.9919] 
+2024-09-10 01:15:01.916890: Epoch time: 340.4 s 
+2024-09-10 01:15:05.483780:  
+2024-09-10 01:15:05.484935: Epoch 686 
+2024-09-10 01:15:05.485422: Current learning rate: 0.00353 
+2024-09-10 01:20:13.200473: train_loss -0.7545 
+2024-09-10 01:20:13.218056: val_loss -0.7448 
+2024-09-10 01:20:13.219244: Pseudo dice [0.9667, 0.9588, 0.9805, 0.9468, 0.8796, 0.9732, 0.9911] 
+2024-09-10 01:20:13.220396: Epoch time: 307.72 s 
+2024-09-10 01:20:17.085012:  
+2024-09-10 01:20:17.086096: Epoch 687 
+2024-09-10 01:20:17.086658: Current learning rate: 0.00352 
+2024-09-10 01:25:15.340430: train_loss -0.7473 
+2024-09-10 01:25:15.357077: val_loss -0.75 
+2024-09-10 01:25:15.358498: Pseudo dice [0.9662, 0.9582, 0.9817, 0.9536, 0.8835, 0.9732, 0.992] 
+2024-09-10 01:25:15.360530: Epoch time: 298.26 s 
+2024-09-10 01:25:19.059854:  
+2024-09-10 01:25:19.067837: Epoch 688 
+2024-09-10 01:25:19.068745: Current learning rate: 0.00351 
+2024-09-10 01:30:32.740366: train_loss -0.7517 
+2024-09-10 01:30:32.757750: val_loss -0.7643 
+2024-09-10 01:30:32.758472: Pseudo dice [0.9668, 0.96, 0.9811, 0.9561, 0.8991, 0.9717, 0.9931] 
+2024-09-10 01:30:32.759062: Epoch time: 313.69 s 
+2024-09-10 01:30:32.759704: Yayy! New best EMA pseudo Dice: 0.9575 
+2024-09-10 01:30:40.976373:  
+2024-09-10 01:30:40.977125: Epoch 689 
+2024-09-10 01:30:40.977599: Current learning rate: 0.0035 
+2024-09-10 01:35:41.269190: train_loss -0.7497 
+2024-09-10 01:35:41.272079: val_loss -0.7371 
+2024-09-10 01:35:41.272789: Pseudo dice [0.9674, 0.9601, 0.9804, 0.955, 0.8774, 0.9721, 0.9902] 
+2024-09-10 01:35:41.273569: Epoch time: 300.3 s 
+2024-09-10 01:35:41.274174: Yayy! New best EMA pseudo Dice: 0.9575 
+2024-09-10 01:35:47.232876:  
+2024-09-10 01:35:47.235120: Epoch 690 
+2024-09-10 01:35:47.236014: Current learning rate: 0.00349 
+2024-09-10 01:40:58.707801: train_loss -0.7525 
+2024-09-10 01:40:58.726016: val_loss -0.7515 
+2024-09-10 01:40:58.727302: Pseudo dice [0.9673, 0.9587, 0.9804, 0.9538, 0.8883, 0.9751, 0.9924] 
+2024-09-10 01:40:58.728222: Epoch time: 311.48 s 
+2024-09-10 01:40:58.729037: Yayy! New best EMA pseudo Dice: 0.9577 
+2024-09-10 01:41:03.957660:  
+2024-09-10 01:41:03.959685: Epoch 691 
+2024-09-10 01:41:03.960304: Current learning rate: 0.00348 
+2024-09-10 01:46:05.423006: train_loss -0.7566 
+2024-09-10 01:46:05.437431: val_loss -0.7424 
+2024-09-10 01:46:05.438417: Pseudo dice [0.9676, 0.9605, 0.9802, 0.9545, 0.8931, 0.9752, 0.9933] 
+2024-09-10 01:46:05.439303: Epoch time: 301.47 s 
+2024-09-10 01:46:05.440109: Yayy! New best EMA pseudo Dice: 0.958 
+2024-09-10 01:46:11.323905:  
+2024-09-10 01:46:11.324744: Epoch 692 
+2024-09-10 01:46:11.325266: Current learning rate: 0.00346 
+2024-09-10 01:51:08.980366: train_loss -0.7515 
+2024-09-10 01:51:08.991356: val_loss -0.725 
+2024-09-10 01:51:08.992049: Pseudo dice [0.9657, 0.961, 0.9758, 0.9455, 0.8725, 0.9711, 0.9882] 
+2024-09-10 01:51:08.992569: Epoch time: 297.66 s 
+2024-09-10 01:51:12.742285:  
+2024-09-10 01:51:12.746354: Epoch 693 
+2024-09-10 01:51:12.747406: Current learning rate: 0.00345 
+2024-09-10 01:56:11.470374: train_loss -0.7586 
+2024-09-10 01:56:11.493537: val_loss -0.7432 
+2024-09-10 01:56:11.494525: Pseudo dice [0.9636, 0.9567, 0.9788, 0.9523, 0.8907, 0.9721, 0.9902] 
+2024-09-10 01:56:11.495476: Epoch time: 298.76 s 
+2024-09-10 01:56:15.671787:  
+2024-09-10 01:56:15.673621: Epoch 694 
+2024-09-10 01:56:15.674832: Current learning rate: 0.00344 
+2024-09-10 02:01:19.900070: train_loss -0.7599 
+2024-09-10 02:01:19.903444: val_loss -0.7573 
+2024-09-10 02:01:19.904361: Pseudo dice [0.9661, 0.9568, 0.9784, 0.9496, 0.8784, 0.9725, 0.9915] 
+2024-09-10 02:01:19.905275: Epoch time: 304.24 s 
+2024-09-10 02:01:22.855867:  
+2024-09-10 02:01:22.861775: Epoch 695 
+2024-09-10 02:01:22.863057: Current learning rate: 0.00343 
+2024-09-10 02:06:19.978373: train_loss -0.7367 
+2024-09-10 02:06:19.981886: val_loss -0.7482 
+2024-09-10 02:06:19.982986: Pseudo dice [0.9635, 0.9583, 0.9762, 0.947, 0.872, 0.9674, 0.9906] 
+2024-09-10 02:06:19.983984: Epoch time: 297.13 s 
+2024-09-10 02:06:22.773354:  
+2024-09-10 02:06:22.775465: Epoch 696 
+2024-09-10 02:06:22.776300: Current learning rate: 0.00342 
+2024-09-10 02:11:26.968598: train_loss -0.7536 
+2024-09-10 02:11:26.982416: val_loss -0.7637 
+2024-09-10 02:11:26.983337: Pseudo dice [0.9662, 0.9583, 0.98, 0.9563, 0.8705, 0.9673, 0.9909] 
+2024-09-10 02:11:26.983988: Epoch time: 304.2 s 
+2024-09-10 02:11:30.186913:  
+2024-09-10 02:11:30.200740: Epoch 697 
+2024-09-10 02:11:30.201854: Current learning rate: 0.00341 
+2024-09-10 02:16:36.315631: train_loss -0.7561 
+2024-09-10 02:16:36.333670: val_loss -0.7667 
+2024-09-10 02:16:36.334540: Pseudo dice [0.9652, 0.9595, 0.9789, 0.9512, 0.8935, 0.9732, 0.993] 
+2024-09-10 02:16:36.335100: Epoch time: 306.14 s 
+2024-09-10 02:16:39.486709:  
+2024-09-10 02:16:39.500395: Epoch 698 
+2024-09-10 02:16:39.501105: Current learning rate: 0.0034 
+2024-09-10 02:22:05.721783: train_loss -0.7423 
+2024-09-10 02:22:05.724572: val_loss -0.7927 
+2024-09-10 02:22:05.725654: Pseudo dice [0.965, 0.9587, 0.9773, 0.9519, 0.8758, 0.9692, 0.9907] 
+2024-09-10 02:22:05.726506: Epoch time: 326.24 s 
+2024-09-10 02:22:09.718465:  
+2024-09-10 02:22:09.719639: Epoch 699 
+2024-09-10 02:22:09.720284: Current learning rate: 0.00339 
+2024-09-10 02:27:21.526678: train_loss -0.7363 
+2024-09-10 02:27:21.553207: val_loss -0.7426 
+2024-09-10 02:27:21.554153: Pseudo dice [0.9653, 0.9587, 0.9782, 0.9492, 0.8927, 0.9708, 0.9912] 
+2024-09-10 02:27:21.555145: Epoch time: 311.81 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/checkpoint_final.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:f520d3293d667986dd89fae115e3602af9f40c98413a08808d1cbb0ce08e074c
+size 819931142

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/debug.json

@@ -0,0 +1,53 @@
+{
+    "_best_ema": "0.9618524118387718",
+    "batch_size": "2",
+    "configuration_manager": "{'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False}",
+    "configuration_name": "3d_fullres",
+    "cudnn_version": 8902,
+    "current_epoch": "1000",
+    "dataloader_train": "<nnunetv2.training.data_augmentation.custom_transforms.limited_length_multithreaded_augmenter.LimitedLenWrapper object at 0x7f310fdf3010>",
+    "dataloader_train.generator": "<nnunetv2.training.dataloading.data_loader_3d.nnUNetDataLoader3D object at 0x7f310fdf3250>",
+    "dataloader_train.num_processes": "16",
+    "dataloader_train.transform": "Compose ( [SpatialTransform( independent_scale_for_each_axis = False, p_rot_per_sample = 0.2, p_scale_per_sample = 0.2, p_el_per_sample = 0, data_key = 'data', label_key = 'seg', patch_size = [192, 192, 192], patch_center_dist_from_border = None, do_elastic_deform = False, alpha = (0, 0), sigma = (0, 0), do_rotation = True, angle_x = (-0.5235987755982988, 0.5235987755982988), angle_y = (-0.5235987755982988, 0.5235987755982988), angle_z = (-0.5235987755982988, 0.5235987755982988), do_scale = True, scale = (0.7, 1.4), border_mode_data = 'constant', border_cval_data = 0, order_data = 3, border_mode_seg = 'constant', border_cval_seg = -1, order_seg = 1, random_crop = False, p_rot_per_axis = 1, p_independent_scale_per_axis = 1 ), GaussianNoiseTransform( p_per_sample = 0.1, data_key = 'data', noise_variance = (0, 0.1), p_per_channel = 1, per_channel = False ), GaussianBlurTransform( p_per_sample = 0.2, different_sigma_per_channel = True, p_per_channel = 0.5, data_key = 'data', blur_sigma = (0.5, 1.0), different_sigma_per_axis = False, p_isotropic = 0 ), BrightnessMultiplicativeTransform( p_per_sample = 0.15, data_key = 'data', multiplier_range = (0.75, 1.25), per_channel = True ), ContrastAugmentationTransform( p_per_sample = 0.15, data_key = 'data', contrast_range = (0.75, 1.25), preserve_range = True, per_channel = True, p_per_channel = 1 ), SimulateLowResolutionTransform( order_upsample = 3, order_downsample = 0, channels = None, per_channel = True, p_per_channel = 0.5, p_per_sample = 0.25, data_key = 'data', zoom_range = (0.5, 1), ignore_axes = None ), GammaTransform( p_per_sample = 0.1, retain_stats = True, per_channel = True, data_key = 'data', gamma_range = (0.7, 1.5), invert_image = True ), GammaTransform( p_per_sample = 0.3, retain_stats = True, per_channel = True, data_key = 'data', gamma_range = (0.7, 1.5), invert_image = False ), MirrorTransform( p_per_sample = 1, data_key = 'data', label_key = 'seg', axes = (0, 1, 2) ), RemoveLabelTransform( output_key = 'seg', input_key = 'seg', replace_with = 0, remove_label = -1 ), RenameTransform( delete_old = True, out_key = 'target', in_key = 'seg' ), DownsampleSegForDSTransform2( axes = None, output_key = 'target', input_key = 'target', order = 0, ds_scales = [[1.0, 1.0, 1.0], [0.5, 0.5, 0.5], [0.25, 0.25, 0.25], [0.125, 0.125, 0.125], [0.0625, 0.0625, 0.0625]] ), NumpyToTensor( keys = ['data', 'target'], cast_to = 'float' )] )",
+    "dataloader_val": "<nnunetv2.training.data_augmentation.custom_transforms.limited_length_multithreaded_augmenter.LimitedLenWrapper object at 0x7f310fdf1c00>",
+    "dataloader_val.generator": "<nnunetv2.training.dataloading.data_loader_3d.nnUNetDataLoader3D object at 0x7f310fdf2f50>",
+    "dataloader_val.num_processes": "8",
+    "dataloader_val.transform": "Compose ( [RemoveLabelTransform( output_key = 'seg', input_key = 'seg', replace_with = 0, remove_label = -1 ), RenameTransform( delete_old = True, out_key = 'target', in_key = 'seg' ), DownsampleSegForDSTransform2( axes = None, output_key = 'target', input_key = 'target', order = 0, ds_scales = [[1.0, 1.0, 1.0], [0.5, 0.5, 0.5], [0.25, 0.25, 0.25], [0.125, 0.125, 0.125], [0.0625, 0.0625, 0.0625]] ), NumpyToTensor( keys = ['data', 'target'], cast_to = 'float' )] )",
+    "dataset_json": "{'channel_names': {'0': 'CT'}, 'labels': {'background': 0, 'muscle': 1, 'bone': 2, 'subcutaneous': 3, 'abdominal': 4, 'thoracic': 5, 'heart': 6, 'lungs': 7}, 'numTraining': 1656, 'file_ending': '.nii.gz', 'dataset_name': 'BodyCompositionCT Compartment Labels v1', 'reference': 'https://doi.org/10.1038/s41598-025-96238-6; https://doi.org/10.1038/s41597-024-03337-6', 'release': '2024-08-29', 'license': 'CC BY 4.0', 'description': 'BodyCompositionCT segmentation labels. Labels 1, 3, 4, and 5 denote anatomical compartments; labels 2, 6, and 7 denote bone, heart, and lungs. The Abdominal and Thoracic compartments exclude organs and larger vessels. IMAT is not a separate output label. Training CTs originated from the SAROS and TotalSegmentator datasets and are not included.'}",
+    "device": "cuda:0",
+    "disable_checkpointing": "False",
+    "enable_deep_supervision": "True",
+    "fold": "1",
+    "folder_with_segs_from_previous_stage": "None",
+    "gpu_name": "NVIDIA A100-SXM4-40GB",
+    "grad_scaler": "<torch.cuda.amp.grad_scaler.GradScaler object at 0x7f309aba0160>",
+    "hostname": "<training-host>",
+    "inference_allowed_mirroring_axes": "(0, 1, 2)",
+    "initial_lr": "0.01",
+    "is_cascaded": "False",
+    "is_ddp": "False",
+    "label_manager": "<nnunetv2.utilities.label_handling.label_handling.LabelManager object at 0x7f309aba02e0>",
+    "local_rank": "0",
+    "log_file": "<workspace>/data/nnUNet_results/Dataset611_Tissue/nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/training_log_2024_9_10_13_25_58.txt",
+    "logger": "<nnunetv2.training.logging.nnunet_logger.nnUNetLogger object at 0x7f309aba01f0>",
+    "loss": "DeepSupervisionWrapper(\n  (loss): DC_and_CE_loss(\n    (ce): RobustCrossEntropyLoss()\n    (dc): MemoryEfficientSoftDiceLoss()\n  )\n)",
+    "lr_scheduler": "<nnunetv2.training.lr_scheduler.polylr.PolyLRScheduler object at 0x7f309ac7d2d0>",
+    "my_init_kwargs": "{'plans': {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'configurations': {'2d': {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 89, 'patch_size': [320, 320], 'median_image_size_in_voxels': [257.0, 260.0], 'spacing': [1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 7, 'features_per_stage': [32, 64, 128, 256, 512, 512, 512], 'conv_op': 'torch.nn.modules.conv.Conv2d', 'kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'strides': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm2d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': True}, '3d_fullres': {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False}}, 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}}, 'configuration': '3d_fullres', 'fold': 1, 'dataset_json': {'channel_names': {'0': 'CT'}, 'labels': {'background': 0, 'muscle': 1, 'bone': 2, 'subcutaneous': 3, 'abdominal': 4, 'thoracic': 5, 'heart': 6, 'lungs': 7}, 'numTraining': 1656, 'file_ending': '.nii.gz', 'dataset_name': 'BodyCompositionCT Compartment Labels v1', 'reference': 'https://doi.org/10.1038/s41598-025-96238-6; https://doi.org/10.1038/s41597-024-03337-6', 'release': '2024-08-29', 'license': 'CC BY 4.0', 'description': 'BodyCompositionCT segmentation labels. Labels 1, 3, 4, and 5 denote anatomical compartments; labels 2, 6, and 7 denote bone, heart, and lungs. The Abdominal and Thoracic compartments exclude organs and larger vessels. IMAT is not a separate output label. Training CTs originated from the SAROS and TotalSegmentator datasets and are not included.'}, 'unpack_dataset': False, 'device': device(type='cuda')}",
+    "network": "OptimizedModule",
+    "num_epochs": "1000",
+    "num_input_channels": "1",
+    "num_iterations_per_epoch": "250",
+    "num_val_iterations_per_epoch": "50",
+    "optimizer": "SGD (\nParameter Group 0\n    dampening: 0\n    differentiable: False\n    foreach: None\n    fused: None\n    initial_lr: 0.01\n    lr: 1.995262314968881e-05\n    maximize: False\n    momentum: 0.99\n    nesterov: True\n    weight_decay: 3e-05\n)",
+    "output_folder": "<workspace>/data/nnUNet_results/Dataset611_Tissue/nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1",
+    "output_folder_base": "<workspace>/data/nnUNet_results/Dataset611_Tissue/nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres",
+    "oversample_foreground_percent": "0.33",
+    "plans_manager": "{'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'configurations': {'2d': {'data_identifier': 'nnUNetPlans_2d', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 89, 'patch_size': [320, 320], 'median_image_size_in_voxels': [257.0, 260.0], 'spacing': [1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 7, 'features_per_stage': [32, 64, 128, 256, 512, 512, 512], 'conv_op': 'torch.nn.modules.conv.Conv2d', 'kernel_sizes': [[3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3], [3, 3]], 'strides': [[1, 1], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2], [2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm2d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': True}, '3d_fullres': {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False}}, 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}}",
+    "preprocessed_dataset_folder": "<workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/nnUNetPlans_3d_fullres",
+    "preprocessed_dataset_folder_base": "<workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue",
+    "save_every": "50",
+    "torch_version": "2.3.0+cu121",
+    "unpack_dataset": "False",
+    "was_initialized": "True",
+    "weight_decay": "3e-05"
+}

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/README.md

@@ -0,0 +1,6 @@
+# Training log note
+
+The fold's `debug.json` names
+`training_log_2024_9_10_13_25_58.txt` as the active nnU-Net log. That file is
+not included; the available timestamped log segments are provided in this
+directory.

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_10_00_29_55.txt

@@ -0,0 +1,731 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-10 00:30:07.902106: Using torch.compile... 
+2024-09-10 00:30:22.458913: do_dummy_2d_data_aug: False 
+2024-09-10 00:30:22.466082: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-10 00:30:22.468709: The split file contains 5 splits. 
+2024-09-10 00:30:22.469503: Desired fold for training: 1 
+2024-09-10 00:30:22.470166: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-10 00:30:32.779784: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-10 00:30:33.676261:  
+2024-09-10 00:30:33.677311: Epoch 850 
+2024-09-10 00:30:33.678330: Current learning rate: 0.00181 
+2024-09-10 00:43:45.141574: train_loss -0.7504 
+2024-09-10 00:43:45.142846: val_loss -0.7925 
+2024-09-10 00:43:45.143560: Pseudo dice [0.9667, 0.9584, 0.9803, 0.9573, 0.8797, 0.976, 0.9897] 
+2024-09-10 00:43:45.144227: Epoch time: 791.47 s 
+2024-09-10 00:43:46.672191:  
+2024-09-10 00:43:46.673070: Epoch 851 
+2024-09-10 00:43:46.673869: Current learning rate: 0.0018 
+2024-09-10 00:48:11.613314: train_loss -0.7599 
+2024-09-10 00:48:11.625389: val_loss -0.7798 
+2024-09-10 00:48:11.626238: Pseudo dice [0.9679, 0.9589, 0.9822, 0.9569, 0.8773, 0.9751, 0.9914] 
+2024-09-10 00:48:11.627036: Epoch time: 264.95 s 
+2024-09-10 00:48:14.180066:  
+2024-09-10 00:48:14.181808: Epoch 852 
+2024-09-10 00:48:14.182657: Current learning rate: 0.00179 
+2024-09-10 00:52:55.435977: train_loss -0.7611 
+2024-09-10 00:52:55.450034: val_loss -0.7356 
+2024-09-10 00:52:55.450975: Pseudo dice [0.9671, 0.9593, 0.9786, 0.9519, 0.8491, 0.959, 0.9859] 
+2024-09-10 00:52:55.451765: Epoch time: 281.26 s 
+2024-09-10 00:52:59.818980:  
+2024-09-10 00:52:59.820045: Epoch 853 
+2024-09-10 00:52:59.821000: Current learning rate: 0.00178 
+2024-09-10 00:57:10.270520: train_loss -0.75 
+2024-09-10 00:57:10.280888: val_loss -0.7477 
+2024-09-10 00:57:10.285181: Pseudo dice [0.9698, 0.9612, 0.9845, 0.9535, 0.8803, 0.9771, 0.9901] 
+2024-09-10 00:57:10.289349: Epoch time: 250.46 s 
+2024-09-10 00:57:13.272086:  
+2024-09-10 00:57:13.284780: Epoch 854 
+2024-09-10 00:57:13.285762: Current learning rate: 0.00177 
+2024-09-10 01:01:18.508296: train_loss -0.7657 
+2024-09-10 01:01:18.510279: val_loss -0.7824 
+2024-09-10 01:01:18.511038: Pseudo dice [0.9695, 0.9602, 0.9822, 0.9558, 0.8816, 0.9761, 0.9905] 
+2024-09-10 01:01:18.511941: Epoch time: 245.24 s 
+2024-09-10 01:01:20.666391:  
+2024-09-10 01:01:20.667335: Epoch 855 
+2024-09-10 01:01:20.668314: Current learning rate: 0.00176 
+2024-09-10 01:05:39.924460: train_loss -0.7514 
+2024-09-10 01:05:39.935322: val_loss -0.7818 
+2024-09-10 01:05:39.936352: Pseudo dice [0.9668, 0.9599, 0.9795, 0.9471, 0.8847, 0.961, 0.9887] 
+2024-09-10 01:05:39.937233: Epoch time: 259.26 s 
+2024-09-10 01:05:42.372734:  
+2024-09-10 01:05:42.373627: Epoch 856 
+2024-09-10 01:05:42.374468: Current learning rate: 0.00175 
+2024-09-10 01:10:06.157843: train_loss -0.7609 
+2024-09-10 01:10:06.160284: val_loss -0.7497 
+2024-09-10 01:10:06.161222: Pseudo dice [0.969, 0.9602, 0.9817, 0.9539, 0.884, 0.96, 0.9895] 
+2024-09-10 01:10:06.162094: Epoch time: 263.79 s 
+2024-09-10 01:10:08.273700:  
+2024-09-10 01:10:08.274942: Epoch 857 
+2024-09-10 01:10:08.276023: Current learning rate: 0.00174 
+2024-09-10 01:14:24.041171: train_loss -0.7595 
+2024-09-10 01:14:24.043435: val_loss -0.6921 
+2024-09-10 01:14:24.044297: Pseudo dice [0.9691, 0.9616, 0.9813, 0.9521, 0.8656, 0.9749, 0.9885] 
+2024-09-10 01:14:24.045099: Epoch time: 255.78 s 
+2024-09-10 01:14:26.328173:  
+2024-09-10 01:14:26.331344: Epoch 858 
+2024-09-10 01:14:26.338712: Current learning rate: 0.00173 
+2024-09-10 01:18:51.933078: train_loss -0.754 
+2024-09-10 01:18:51.953599: val_loss -0.7198 
+2024-09-10 01:18:51.954451: Pseudo dice [0.9685, 0.9608, 0.9801, 0.9563, 0.8862, 0.9707, 0.9897] 
+2024-09-10 01:18:51.955248: Epoch time: 265.62 s 
+2024-09-10 01:18:54.112124:  
+2024-09-10 01:18:54.121661: Epoch 859 
+2024-09-10 01:18:54.122658: Current learning rate: 0.00172 
+2024-09-10 01:22:58.549546: train_loss -0.7631 
+2024-09-10 01:22:58.566025: val_loss -0.7491 
+2024-09-10 01:22:58.567056: Pseudo dice [0.9688, 0.9632, 0.9811, 0.9549, 0.8905, 0.9757, 0.9919] 
+2024-09-10 01:22:58.567885: Epoch time: 244.44 s 
+2024-09-10 01:23:00.633244:  
+2024-09-10 01:23:00.634444: Epoch 860 
+2024-09-10 01:23:00.635361: Current learning rate: 0.0017 
+2024-09-10 01:27:21.785263: train_loss -0.759 
+2024-09-10 01:27:21.802197: val_loss -0.7657 
+2024-09-10 01:27:21.803106: Pseudo dice [0.9673, 0.9611, 0.9816, 0.9559, 0.881, 0.9761, 0.9899] 
+2024-09-10 01:27:21.803934: Epoch time: 261.16 s 
+2024-09-10 01:27:24.187389:  
+2024-09-10 01:27:24.189377: Epoch 861 
+2024-09-10 01:27:24.190260: Current learning rate: 0.00169 
+2024-09-10 01:31:31.529446: train_loss -0.7671 
+2024-09-10 01:31:31.557709: val_loss -0.7178 
+2024-09-10 01:31:31.558642: Pseudo dice [0.9683, 0.9627, 0.98, 0.9522, 0.8602, 0.9584, 0.9852] 
+2024-09-10 01:31:31.559443: Epoch time: 247.35 s 
+2024-09-10 01:31:33.676797:  
+2024-09-10 01:31:33.678420: Epoch 862 
+2024-09-10 01:31:33.679239: Current learning rate: 0.00168 
+2024-09-10 01:36:00.097215: train_loss -0.7656 
+2024-09-10 01:36:00.099515: val_loss -0.7345 
+2024-09-10 01:36:00.100303: Pseudo dice [0.9665, 0.9599, 0.9795, 0.9524, 0.8749, 0.9706, 0.9896] 
+2024-09-10 01:36:00.101201: Epoch time: 266.43 s 
+2024-09-10 01:36:02.433436:  
+2024-09-10 01:36:02.436891: Epoch 863 
+2024-09-10 01:36:02.448588: Current learning rate: 0.00167 
+2024-09-10 01:40:21.986691: train_loss -0.7667 
+2024-09-10 01:40:21.989348: val_loss -0.7622 
+2024-09-10 01:40:21.990220: Pseudo dice [0.97, 0.9629, 0.9814, 0.9526, 0.8871, 0.9756, 0.9906] 
+2024-09-10 01:40:21.991002: Epoch time: 259.56 s 
+2024-09-10 01:40:24.174196:  
+2024-09-10 01:40:24.175406: Epoch 864 
+2024-09-10 01:40:24.176223: Current learning rate: 0.00166 
+2024-09-10 01:44:41.988790: train_loss -0.7622 
+2024-09-10 01:44:41.990226: val_loss -0.7461 
+2024-09-10 01:44:41.991010: Pseudo dice [0.9692, 0.9633, 0.9823, 0.9529, 0.8729, 0.9765, 0.9907] 
+2024-09-10 01:44:41.991743: Epoch time: 257.82 s 
+2024-09-10 01:44:44.214972:  
+2024-09-10 01:44:44.216231: Epoch 865 
+2024-09-10 01:44:44.217420: Current learning rate: 0.00165 
+2024-09-10 01:49:07.746939: train_loss -0.7617 
+2024-09-10 01:49:07.761455: val_loss -0.7303 
+2024-09-10 01:49:07.762344: Pseudo dice [0.9661, 0.958, 0.9779, 0.9533, 0.8554, 0.948, 0.9813] 
+2024-09-10 01:49:07.763420: Epoch time: 263.54 s 
+2024-09-10 01:49:10.073784:  
+2024-09-10 01:49:10.074992: Epoch 866 
+2024-09-10 01:49:10.075902: Current learning rate: 0.00164 
+2024-09-10 01:53:49.011041: train_loss -0.754 
+2024-09-10 01:53:49.025340: val_loss -0.717 
+2024-09-10 01:53:49.026263: Pseudo dice [0.9684, 0.9594, 0.9806, 0.9527, 0.8656, 0.9624, 0.9878] 
+2024-09-10 01:53:49.027184: Epoch time: 278.94 s 
+2024-09-10 01:53:51.118798:  
+2024-09-10 01:53:51.119724: Epoch 867 
+2024-09-10 01:53:51.123022: Current learning rate: 0.00163 
+2024-09-10 01:58:22.343446: train_loss -0.7594 
+2024-09-10 01:58:22.358111: val_loss -0.7753 
+2024-09-10 01:58:22.359041: Pseudo dice [0.962, 0.961, 0.9749, 0.9522, 0.8569, 0.9738, 0.9838] 
+2024-09-10 01:58:22.359849: Epoch time: 271.23 s 
+2024-09-10 01:58:24.564665:  
+2024-09-10 01:58:24.566299: Epoch 868 
+2024-09-10 01:58:24.568919: Current learning rate: 0.00162 
+2024-09-10 02:02:50.486759: train_loss -0.7596 
+2024-09-10 02:02:50.501134: val_loss -0.7543 
+2024-09-10 02:02:50.502156: Pseudo dice [0.9664, 0.9594, 0.9797, 0.9529, 0.8897, 0.9749, 0.9908] 
+2024-09-10 02:02:50.502981: Epoch time: 265.93 s 
+2024-09-10 02:02:54.336710:  
+2024-09-10 02:02:54.337740: Epoch 869 
+2024-09-10 02:02:54.338579: Current learning rate: 0.00161 
+2024-09-10 02:07:11.461520: train_loss -0.7523 
+2024-09-10 02:07:11.477249: val_loss -0.7296 
+2024-09-10 02:07:11.478138: Pseudo dice [0.9666, 0.9583, 0.9788, 0.9483, 0.8786, 0.9773, 0.9906] 
+2024-09-10 02:07:11.478918: Epoch time: 257.13 s 
+2024-09-10 02:07:14.621212:  
+2024-09-10 02:07:14.637511: Epoch 870 
+2024-09-10 02:07:14.638369: Current learning rate: 0.00159 
+2024-09-10 02:11:47.473406: train_loss -0.7619 
+2024-09-10 02:11:47.474868: val_loss -0.7818 
+2024-09-10 02:11:47.475775: Pseudo dice [0.9696, 0.9606, 0.9813, 0.9538, 0.8932, 0.9671, 0.9901] 
+2024-09-10 02:11:47.476611: Epoch time: 272.87 s 
+2024-09-10 02:11:49.700612:  
+2024-09-10 02:11:49.701663: Epoch 871 
+2024-09-10 02:11:49.702579: Current learning rate: 0.00158 
+2024-09-10 02:15:56.930458: train_loss -0.7486 
+2024-09-10 02:15:56.932179: val_loss -0.7661 
+2024-09-10 02:15:56.933134: Pseudo dice [0.9688, 0.9612, 0.9839, 0.9547, 0.8806, 0.9748, 0.9897] 
+2024-09-10 02:15:56.933980: Epoch time: 247.24 s 
+2024-09-10 02:15:58.909010:  
+2024-09-10 02:15:58.910240: Epoch 872 
+2024-09-10 02:15:58.911097: Current learning rate: 0.00157 
+2024-09-10 02:20:18.618672: train_loss -0.748 
+2024-09-10 02:20:18.619946: val_loss -0.7517 
+2024-09-10 02:20:18.620711: Pseudo dice [0.9669, 0.9614, 0.9782, 0.9549, 0.8826, 0.9757, 0.9915] 
+2024-09-10 02:20:18.621583: Epoch time: 259.72 s 
+2024-09-10 02:20:20.953693:  
+2024-09-10 02:20:20.954798: Epoch 873 
+2024-09-10 02:20:20.955734: Current learning rate: 0.00156 
+2024-09-10 02:25:09.596462: train_loss -0.7567 
+2024-09-10 02:25:09.612190: val_loss -0.7505 
+2024-09-10 02:25:09.613248: Pseudo dice [0.9656, 0.9592, 0.9787, 0.9531, 0.8777, 0.9624, 0.9868] 
+2024-09-10 02:25:09.614091: Epoch time: 288.65 s 
+2024-09-10 02:25:11.676674:  
+2024-09-10 02:25:11.688684: Epoch 874 
+2024-09-10 02:25:11.689497: Current learning rate: 0.00155 
+2024-09-10 02:29:43.582289: train_loss -0.7548 
+2024-09-10 02:29:43.602155: val_loss -0.7529 
+2024-09-10 02:29:43.603242: Pseudo dice [0.9664, 0.9606, 0.9798, 0.9578, 0.8919, 0.9747, 0.9914] 
+2024-09-10 02:29:43.604176: Epoch time: 271.92 s 
+2024-09-10 02:29:47.989177:  
+2024-09-10 02:29:48.000740: Epoch 875 
+2024-09-10 02:29:48.001482: Current learning rate: 0.00154 
+2024-09-10 02:34:13.660772: train_loss -0.7597 
+2024-09-10 02:34:13.663220: val_loss -0.8149 
+2024-09-10 02:34:13.664198: Pseudo dice [0.9685, 0.9598, 0.9818, 0.9567, 0.8863, 0.9741, 0.9912] 
+2024-09-10 02:34:13.665115: Epoch time: 265.68 s 
+2024-09-10 02:34:15.950113:  
+2024-09-10 02:34:15.951347: Epoch 876 
+2024-09-10 02:34:15.952237: Current learning rate: 0.00153 
+2024-09-10 02:38:51.010022: train_loss -0.7407 
+2024-09-10 02:38:51.012499: val_loss -0.7973 
+2024-09-10 02:38:51.013367: Pseudo dice [0.9664, 0.9575, 0.9808, 0.946, 0.8761, 0.9755, 0.9925] 
+2024-09-10 02:38:51.014305: Epoch time: 275.06 s 
+2024-09-10 02:38:53.300185:  
+2024-09-10 02:38:53.301195: Epoch 877 
+2024-09-10 02:38:53.302009: Current learning rate: 0.00152 
+2024-09-10 02:43:17.090586: train_loss -0.7568 
+2024-09-10 02:43:17.109500: val_loss -0.7566 
+2024-09-10 02:43:17.110895: Pseudo dice [0.9687, 0.9622, 0.9802, 0.9574, 0.8942, 0.9629, 0.9886] 
+2024-09-10 02:43:17.112324: Epoch time: 263.8 s 
+2024-09-10 02:43:19.248592:  
+2024-09-10 02:43:19.249783: Epoch 878 
+2024-09-10 02:43:19.250557: Current learning rate: 0.00151 
+2024-09-10 02:47:34.828390: train_loss -0.7563 
+2024-09-10 02:47:34.830472: val_loss -0.7238 
+2024-09-10 02:47:34.831289: Pseudo dice [0.9693, 0.9623, 0.9815, 0.9561, 0.8596, 0.9612, 0.9851] 
+2024-09-10 02:47:34.832108: Epoch time: 255.58 s 
+2024-09-10 02:47:37.685658:  
+2024-09-10 02:47:37.700762: Epoch 879 
+2024-09-10 02:47:37.701792: Current learning rate: 0.00149 
+2024-09-10 02:51:53.467058: train_loss -0.7569 
+2024-09-10 02:51:53.469382: val_loss -0.7865 
+2024-09-10 02:51:53.470249: Pseudo dice [0.9668, 0.9602, 0.9798, 0.9494, 0.8778, 0.9732, 0.9885] 
+2024-09-10 02:51:53.471129: Epoch time: 255.79 s 
+2024-09-10 02:51:57.234113:  
+2024-09-10 02:51:57.235025: Epoch 880 
+2024-09-10 02:51:57.235795: Current learning rate: 0.00148 
+2024-09-10 02:56:15.767797: train_loss -0.7594 
+2024-09-10 02:56:15.774634: val_loss -0.7546 
+2024-09-10 02:56:15.776584: Pseudo dice [0.9667, 0.9617, 0.9772, 0.9602, 0.8861, 0.9708, 0.9912] 
+2024-09-10 02:56:15.778902: Epoch time: 258.54 s 
+2024-09-10 02:56:18.134292:  
+2024-09-10 02:56:18.135299: Epoch 881 
+2024-09-10 02:56:18.136040: Current learning rate: 0.00147 
+2024-09-10 03:00:36.575429: train_loss -0.7618 
+2024-09-10 03:00:36.586568: val_loss -0.7809 
+2024-09-10 03:00:36.587639: Pseudo dice [0.9695, 0.9613, 0.9822, 0.9552, 0.8968, 0.9731, 0.9923] 
+2024-09-10 03:00:36.591253: Epoch time: 258.45 s 
+2024-09-10 03:00:40.521236:  
+2024-09-10 03:00:40.524994: Epoch 882 
+2024-09-10 03:00:40.525887: Current learning rate: 0.00146 
+2024-09-10 03:04:57.311944: train_loss -0.7539 
+2024-09-10 03:04:57.326109: val_loss -0.7159 
+2024-09-10 03:04:57.326982: Pseudo dice [0.9662, 0.9576, 0.9781, 0.9542, 0.8598, 0.9515, 0.982] 
+2024-09-10 03:04:57.327767: Epoch time: 256.8 s 
+2024-09-10 03:05:00.497779:  
+2024-09-10 03:05:00.516117: Epoch 883 
+2024-09-10 03:05:00.517289: Current learning rate: 0.00145 
+2024-09-10 03:09:17.043113: train_loss -0.7579 
+2024-09-10 03:09:17.061356: val_loss -0.7527 
+2024-09-10 03:09:17.062274: Pseudo dice [0.9675, 0.9613, 0.9778, 0.9562, 0.8616, 0.9665, 0.9869] 
+2024-09-10 03:09:17.063104: Epoch time: 256.55 s 
+2024-09-10 03:09:19.200767:  
+2024-09-10 03:09:19.202165: Epoch 884 
+2024-09-10 03:09:19.203156: Current learning rate: 0.00144 
+2024-09-10 03:13:41.885376: train_loss -0.7527 
+2024-09-10 03:13:41.887939: val_loss -0.722 
+2024-09-10 03:13:41.888900: Pseudo dice [0.9661, 0.9606, 0.9784, 0.9549, 0.8852, 0.9575, 0.9882] 
+2024-09-10 03:13:41.889828: Epoch time: 262.69 s 
+2024-09-10 03:13:44.061514:  
+2024-09-10 03:13:44.062622: Epoch 885 
+2024-09-10 03:13:44.063424: Current learning rate: 0.00143 
+2024-09-10 03:18:10.171670: train_loss -0.7533 
+2024-09-10 03:18:10.182597: val_loss -0.7552 
+2024-09-10 03:18:10.183532: Pseudo dice [0.9695, 0.9627, 0.9821, 0.9579, 0.9028, 0.975, 0.993] 
+2024-09-10 03:18:10.184234: Epoch time: 266.11 s 
+2024-09-10 03:18:13.228441:  
+2024-09-10 03:18:13.229393: Epoch 886 
+2024-09-10 03:18:13.230224: Current learning rate: 0.00142 
+2024-09-10 03:22:31.323687: train_loss -0.7629 
+2024-09-10 03:22:31.326230: val_loss -0.7498 
+2024-09-10 03:22:31.327191: Pseudo dice [0.9674, 0.9614, 0.9803, 0.9537, 0.8775, 0.9738, 0.9897] 
+2024-09-10 03:22:31.328079: Epoch time: 258.1 s 
+2024-09-10 03:22:33.883821:  
+2024-09-10 03:22:33.884704: Epoch 887 
+2024-09-10 03:22:33.885330: Current learning rate: 0.00141 
+2024-09-10 03:26:58.595236: train_loss -0.7712 
+2024-09-10 03:26:58.620865: val_loss -0.7656 
+2024-09-10 03:26:58.621986: Pseudo dice [0.9701, 0.9611, 0.9831, 0.9588, 0.8864, 0.9733, 0.9915] 
+2024-09-10 03:26:58.622836: Epoch time: 264.72 s 
+2024-09-10 03:27:00.781526:  
+2024-09-10 03:27:00.782727: Epoch 888 
+2024-09-10 03:27:00.783791: Current learning rate: 0.00139 
+2024-09-10 03:31:24.825476: train_loss -0.7582 
+2024-09-10 03:31:24.827739: val_loss -0.7553 
+2024-09-10 03:31:24.828800: Pseudo dice [0.9686, 0.9598, 0.9805, 0.9558, 0.8877, 0.964, 0.9903] 
+2024-09-10 03:31:24.829726: Epoch time: 264.05 s 
+2024-09-10 03:31:26.908785:  
+2024-09-10 03:31:26.910093: Epoch 889 
+2024-09-10 03:31:26.910991: Current learning rate: 0.00138 
+2024-09-10 03:35:42.705982: train_loss -0.7493 
+2024-09-10 03:35:42.707609: val_loss -0.7676 
+2024-09-10 03:35:42.708676: Pseudo dice [0.9673, 0.9607, 0.9797, 0.9553, 0.9011, 0.9757, 0.9925] 
+2024-09-10 03:35:42.709754: Epoch time: 255.81 s 
+2024-09-10 03:35:42.710759: Yayy! New best EMA pseudo Dice: 0.958 
+2024-09-10 03:35:48.973598:  
+2024-09-10 03:35:48.976108: Epoch 890 
+2024-09-10 03:35:48.980395: Current learning rate: 0.00137 
+2024-09-10 03:40:09.276987: train_loss -0.7548 
+2024-09-10 03:40:09.278677: val_loss -0.7886 
+2024-09-10 03:40:09.279812: Pseudo dice [0.9691, 0.9606, 0.9835, 0.9529, 0.8403, 0.9757, 0.9866] 
+2024-09-10 03:40:09.292748: Epoch time: 260.31 s 
+2024-09-10 03:40:12.176678:  
+2024-09-10 03:40:12.178038: Epoch 891 
+2024-09-10 03:40:12.179152: Current learning rate: 0.00136 
+2024-09-10 03:44:38.864587: train_loss -0.7509 
+2024-09-10 03:44:38.865926: val_loss -0.7811 
+2024-09-10 03:44:38.866769: Pseudo dice [0.9677, 0.9613, 0.9791, 0.9519, 0.8897, 0.9773, 0.9928] 
+2024-09-10 03:44:38.867617: Epoch time: 266.69 s 
+2024-09-10 03:44:41.203419:  
+2024-09-10 03:44:41.204440: Epoch 892 
+2024-09-10 03:44:41.205282: Current learning rate: 0.00135 
+2024-09-10 03:48:58.814748: train_loss -0.7455 
+2024-09-10 03:48:58.817049: val_loss -0.7432 
+2024-09-10 03:48:58.817895: Pseudo dice [0.9687, 0.9615, 0.9807, 0.9569, 0.8879, 0.9634, 0.9891] 
+2024-09-10 03:48:58.818647: Epoch time: 257.62 s 
+2024-09-10 03:49:02.294158:  
+2024-09-10 03:49:02.295355: Epoch 893 
+2024-09-10 03:49:02.296181: Current learning rate: 0.00134 
+2024-09-10 03:53:27.790557: train_loss -0.749 
+2024-09-10 03:53:27.809851: val_loss -0.7731 
+2024-09-10 03:53:27.810784: Pseudo dice [0.9679, 0.96, 0.9792, 0.9528, 0.8787, 0.9734, 0.9902] 
+2024-09-10 03:53:27.811588: Epoch time: 265.5 s 
+2024-09-10 03:53:30.996159:  
+2024-09-10 03:53:30.997283: Epoch 894 
+2024-09-10 03:53:30.998427: Current learning rate: 0.00133 
+2024-09-10 03:57:53.317465: train_loss -0.7659 
+2024-09-10 03:57:53.333199: val_loss -0.7261 
+2024-09-10 03:57:53.334030: Pseudo dice [0.9685, 0.9603, 0.9802, 0.9564, 0.8605, 0.9497, 0.9834] 
+2024-09-10 03:57:53.334851: Epoch time: 262.33 s 
+2024-09-10 03:57:55.495175:  
+2024-09-10 03:57:55.496176: Epoch 895 
+2024-09-10 03:57:55.497066: Current learning rate: 0.00132 
+2024-09-10 04:01:58.829972: train_loss -0.7647 
+2024-09-10 04:01:58.832467: val_loss -0.7826 
+2024-09-10 04:01:58.833530: Pseudo dice [0.968, 0.9608, 0.9822, 0.9534, 0.8699, 0.9759, 0.9875] 
+2024-09-10 04:01:58.834542: Epoch time: 243.34 s 
+2024-09-10 04:02:02.189770:  
+2024-09-10 04:02:02.190847: Epoch 896 
+2024-09-10 04:02:02.191798: Current learning rate: 0.0013 
+2024-09-10 04:06:30.920287: train_loss -0.7784 
+2024-09-10 04:06:30.922795: val_loss -0.77 
+2024-09-10 04:06:30.923685: Pseudo dice [0.9689, 0.9595, 0.9832, 0.9556, 0.9056, 0.966, 0.9899] 
+2024-09-10 04:06:30.924634: Epoch time: 268.74 s 
+2024-09-10 04:06:33.169254:  
+2024-09-10 04:06:33.170517: Epoch 897 
+2024-09-10 04:06:33.171382: Current learning rate: 0.00129 
+2024-09-10 04:10:38.168092: train_loss -0.7611 
+2024-09-10 04:10:38.170607: val_loss -0.7746 
+2024-09-10 04:10:38.171612: Pseudo dice [0.9689, 0.9634, 0.9819, 0.9562, 0.8889, 0.975, 0.9907] 
+2024-09-10 04:10:38.172555: Epoch time: 245.0 s 
+2024-09-10 04:10:40.221392:  
+2024-09-10 04:10:40.222494: Epoch 898 
+2024-09-10 04:10:40.223433: Current learning rate: 0.00128 
+2024-09-10 04:15:07.650064: train_loss -0.7552 
+2024-09-10 04:15:07.652351: val_loss -0.7626 
+2024-09-10 04:15:07.653211: Pseudo dice [0.9686, 0.9603, 0.981, 0.9537, 0.8941, 0.9759, 0.9931] 
+2024-09-10 04:15:07.654010: Epoch time: 267.43 s 
+2024-09-10 04:15:07.654846: Yayy! New best EMA pseudo Dice: 0.9581 
+2024-09-10 04:15:13.247589:  
+2024-09-10 04:15:13.248498: Epoch 899 
+2024-09-10 04:15:13.249266: Current learning rate: 0.00127 
+2024-09-10 04:19:31.158303: train_loss -0.7617 
+2024-09-10 04:19:31.160720: val_loss -0.7182 
+2024-09-10 04:19:31.161535: Pseudo dice [0.9667, 0.958, 0.9778, 0.9519, 0.8747, 0.9635, 0.9866] 
+2024-09-10 04:19:31.162355: Epoch time: 257.91 s 
+2024-09-10 04:19:38.321789:  
+2024-09-10 04:19:38.322526: Epoch 900 
+2024-09-10 04:19:38.324923: Current learning rate: 0.00126 
+2024-09-10 04:24:04.173394: train_loss -0.7589 
+2024-09-10 04:24:04.190137: val_loss -0.778 
+2024-09-10 04:24:04.190985: Pseudo dice [0.969, 0.9605, 0.9818, 0.9549, 0.8915, 0.9763, 0.9913] 
+2024-09-10 04:24:04.191741: Epoch time: 265.85 s 
+2024-09-10 04:24:06.451766:  
+2024-09-10 04:24:06.452797: Epoch 901 
+2024-09-10 04:24:06.453756: Current learning rate: 0.00125 
+2024-09-10 04:28:41.069669: train_loss -0.7382 
+2024-09-10 04:28:41.071850: val_loss -0.7451 
+2024-09-10 04:28:41.072718: Pseudo dice [0.9666, 0.9632, 0.9778, 0.9558, 0.8998, 0.9572, 0.9895] 
+2024-09-10 04:28:41.073478: Epoch time: 274.62 s 
+2024-09-10 04:28:43.188509:  
+2024-09-10 04:28:43.189820: Epoch 902 
+2024-09-10 04:28:43.190733: Current learning rate: 0.00124 
+2024-09-10 04:33:06.425782: train_loss -0.7597 
+2024-09-10 04:33:06.428336: val_loss -0.76 
+2024-09-10 04:33:06.429282: Pseudo dice [0.9698, 0.9613, 0.9827, 0.9542, 0.8772, 0.9751, 0.9877] 
+2024-09-10 04:33:06.430074: Epoch time: 263.24 s 
+2024-09-10 04:33:08.655818:  
+2024-09-10 04:33:08.657154: Epoch 903 
+2024-09-10 04:33:08.657959: Current learning rate: 0.00122 
+2024-09-10 04:37:31.026425: train_loss -0.7599 
+2024-09-10 04:37:31.028894: val_loss -0.7551 
+2024-09-10 04:37:31.030082: Pseudo dice [0.969, 0.9611, 0.9803, 0.9512, 0.8759, 0.9744, 0.9903] 
+2024-09-10 04:37:31.031134: Epoch time: 262.38 s 
+2024-09-10 04:37:33.043823:  
+2024-09-10 04:37:33.044871: Epoch 904 
+2024-09-10 04:37:33.045707: Current learning rate: 0.00121 
+2024-09-10 04:41:55.533458: train_loss -0.7496 
+2024-09-10 04:41:55.556180: val_loss -0.7535 
+2024-09-10 04:41:55.557327: Pseudo dice [0.9666, 0.9605, 0.9793, 0.9541, 0.8883, 0.9514, 0.987] 
+2024-09-10 04:41:55.558352: Epoch time: 262.49 s 
+2024-09-10 04:41:57.946866:  
+2024-09-10 04:41:57.948556: Epoch 905 
+2024-09-10 04:41:57.949742: Current learning rate: 0.0012 
+2024-09-10 04:46:26.018152: train_loss -0.7587 
+2024-09-10 04:46:26.030154: val_loss -0.7808 
+2024-09-10 04:46:26.030950: Pseudo dice [0.9661, 0.9606, 0.9789, 0.9538, 0.891, 0.9751, 0.9913] 
+2024-09-10 04:46:26.031688: Epoch time: 268.08 s 
+2024-09-10 04:46:30.450604:  
+2024-09-10 04:46:30.451826: Epoch 906 
+2024-09-10 04:46:30.452865: Current learning rate: 0.00119 
+2024-09-10 04:50:33.813455: train_loss -0.7569 
+2024-09-10 04:50:33.815964: val_loss -0.7275 
+2024-09-10 04:50:33.822151: Pseudo dice [0.9695, 0.961, 0.9822, 0.9543, 0.8668, 0.9617, 0.9881] 
+2024-09-10 04:50:33.823282: Epoch time: 243.37 s 
+2024-09-10 04:50:35.884213:  
+2024-09-10 04:50:35.896740: Epoch 907 
+2024-09-10 04:50:35.897987: Current learning rate: 0.00118 
+2024-09-10 04:54:56.645038: train_loss -0.7635 
+2024-09-10 04:54:56.647342: val_loss -0.7842 
+2024-09-10 04:54:56.652108: Pseudo dice [0.9686, 0.961, 0.9817, 0.9551, 0.8844, 0.9759, 0.9898] 
+2024-09-10 04:54:56.664918: Epoch time: 260.77 s 
+2024-09-10 04:54:58.833936:  
+2024-09-10 04:54:58.848763: Epoch 908 
+2024-09-10 04:54:58.849743: Current learning rate: 0.00117 
+2024-09-10 04:59:33.980863: train_loss -0.7505 
+2024-09-10 04:59:33.983251: val_loss -0.7604 
+2024-09-10 04:59:33.984030: Pseudo dice [0.9688, 0.9619, 0.9825, 0.9534, 0.8673, 0.9751, 0.9899] 
+2024-09-10 04:59:33.984976: Epoch time: 275.15 s 
+2024-09-10 04:59:36.268055:  
+2024-09-10 04:59:36.269098: Epoch 909 
+2024-09-10 04:59:36.269996: Current learning rate: 0.00116 
+2024-09-10 05:03:45.116561: train_loss -0.7582 
+2024-09-10 05:03:45.118853: val_loss -0.7909 
+2024-09-10 05:03:45.119697: Pseudo dice [0.9655, 0.9611, 0.9765, 0.9512, 0.9003, 0.9767, 0.9929] 
+2024-09-10 05:03:45.120602: Epoch time: 248.85 s 
+2024-09-10 05:03:47.161979:  
+2024-09-10 05:03:47.163217: Epoch 910 
+2024-09-10 05:03:47.163918: Current learning rate: 0.00115 
+2024-09-10 05:08:03.477975: train_loss -0.7561 
+2024-09-10 05:08:03.479637: val_loss -0.7386 
+2024-09-10 05:08:03.481359: Pseudo dice [0.9666, 0.9601, 0.9805, 0.9487, 0.8791, 0.9748, 0.9904] 
+2024-09-10 05:08:03.482060: Epoch time: 256.32 s 
+2024-09-10 05:08:05.351092:  
+2024-09-10 05:08:05.352289: Epoch 911 
+2024-09-10 05:08:05.353424: Current learning rate: 0.00113 
+2024-09-10 05:12:26.923873: train_loss -0.7592 
+2024-09-10 05:12:26.925558: val_loss -0.7658 
+2024-09-10 05:12:26.926535: Pseudo dice [0.9665, 0.9605, 0.9793, 0.9565, 0.8794, 0.9632, 0.9877] 
+2024-09-10 05:12:26.927413: Epoch time: 261.59 s 
+2024-09-10 05:12:29.032792:  
+2024-09-10 05:12:29.044599: Epoch 912 
+2024-09-10 05:12:29.056832: Current learning rate: 0.00112 
+2024-09-10 05:16:50.071507: train_loss -0.7634 
+2024-09-10 05:16:50.073029: val_loss -0.7864 
+2024-09-10 05:16:50.074008: Pseudo dice [0.9697, 0.9618, 0.984, 0.956, 0.904, 0.9764, 0.9931] 
+2024-09-10 05:16:50.075016: Epoch time: 261.04 s 
+2024-09-10 05:16:50.075892: Yayy! New best EMA pseudo Dice: 0.9583 
+2024-09-10 05:16:57.202446:  
+2024-09-10 05:16:57.203416: Epoch 913 
+2024-09-10 05:16:57.204243: Current learning rate: 0.00111 
+2024-09-10 05:21:12.825587: train_loss -0.7696 
+2024-09-10 05:21:12.827066: val_loss -0.756 
+2024-09-10 05:21:12.828078: Pseudo dice [0.9678, 0.9599, 0.9816, 0.9526, 0.8906, 0.9716, 0.991] 
+2024-09-10 05:21:12.828930: Epoch time: 255.63 s 
+2024-09-10 05:21:12.829733: Yayy! New best EMA pseudo Dice: 0.9584 
+2024-09-10 05:21:19.460657:  
+2024-09-10 05:21:19.461687: Epoch 914 
+2024-09-10 05:21:19.462525: Current learning rate: 0.0011 
+2024-09-10 05:25:58.018361: train_loss -0.7583 
+2024-09-10 05:25:58.027924: val_loss -0.7594 
+2024-09-10 05:25:58.044861: Pseudo dice [0.9698, 0.9627, 0.9817, 0.9597, 0.8954, 0.9621, 0.9888] 
+2024-09-10 05:25:58.045880: Epoch time: 278.56 s 
+2024-09-10 05:25:58.046677: Yayy! New best EMA pseudo Dice: 0.9586 
+2024-09-10 05:26:04.459800:  
+2024-09-10 05:26:04.460850: Epoch 915 
+2024-09-10 05:26:04.461690: Current learning rate: 0.00109 
+2024-09-10 05:30:27.661016: train_loss -0.7553 
+2024-09-10 05:30:27.668194: val_loss -0.7605 
+2024-09-10 05:30:27.670716: Pseudo dice [0.9681, 0.9608, 0.981, 0.9574, 0.8474, 0.9764, 0.9859] 
+2024-09-10 05:30:27.672872: Epoch time: 263.21 s 
+2024-09-10 05:30:30.039252:  
+2024-09-10 05:30:30.040310: Epoch 916 
+2024-09-10 05:30:30.041274: Current learning rate: 0.00108 
+2024-09-10 05:35:02.460812: train_loss -0.7666 
+2024-09-10 05:35:02.462050: val_loss -0.7824 
+2024-09-10 05:35:02.462851: Pseudo dice [0.9659, 0.9578, 0.9801, 0.953, 0.8876, 0.9638, 0.989] 
+2024-09-10 05:35:02.463612: Epoch time: 272.43 s 
+2024-09-10 05:35:05.461330:  
+2024-09-10 05:35:05.462343: Epoch 917 
+2024-09-10 05:35:05.463233: Current learning rate: 0.00106 
+2024-09-10 05:39:19.596706: train_loss -0.7557 
+2024-09-10 05:39:19.598334: val_loss -0.7518 
+2024-09-10 05:39:19.599213: Pseudo dice [0.968, 0.9636, 0.9797, 0.9556, 0.8983, 0.9553, 0.9882] 
+2024-09-10 05:39:19.616211: Epoch time: 254.14 s 
+2024-09-10 05:39:22.796942:  
+2024-09-10 05:39:22.798033: Epoch 918 
+2024-09-10 05:39:22.798856: Current learning rate: 0.00105 
+2024-09-10 05:43:48.286592: train_loss -0.7457 
+2024-09-10 05:43:48.288489: val_loss -0.7574 
+2024-09-10 05:43:48.289824: Pseudo dice [0.9665, 0.9593, 0.9796, 0.9551, 0.8772, 0.9764, 0.9892] 
+2024-09-10 05:43:48.291094: Epoch time: 265.49 s 
+2024-09-10 05:43:51.770049:  
+2024-09-10 05:43:51.796050: Epoch 919 
+2024-09-10 05:43:51.800709: Current learning rate: 0.00104 
+2024-09-10 05:48:04.092755: train_loss -0.758 
+2024-09-10 05:48:04.095180: val_loss -0.7825 
+2024-09-10 05:48:04.095948: Pseudo dice [0.9685, 0.9595, 0.9818, 0.9574, 0.8929, 0.9763, 0.9923] 
+2024-09-10 05:48:04.096796: Epoch time: 252.33 s 
+2024-09-10 05:48:06.173418:  
+2024-09-10 05:48:06.174655: Epoch 920 
+2024-09-10 05:48:06.175384: Current learning rate: 0.00103 
+2024-09-10 05:52:32.620292: train_loss -0.7636 
+2024-09-10 05:52:32.622651: val_loss -0.741 
+2024-09-10 05:52:32.623495: Pseudo dice [0.9652, 0.9574, 0.9769, 0.9452, 0.8795, 0.9639, 0.9876] 
+2024-09-10 05:52:32.624316: Epoch time: 266.45 s 
+2024-09-10 05:52:34.792922:  
+2024-09-10 05:52:34.793904: Epoch 921 
+2024-09-10 05:52:34.794754: Current learning rate: 0.00102 
+2024-09-10 05:56:49.243024: train_loss -0.7526 
+2024-09-10 05:56:49.246147: val_loss -0.7647 
+2024-09-10 05:56:49.247461: Pseudo dice [0.968, 0.962, 0.9813, 0.9586, 0.8867, 0.9752, 0.9912] 
+2024-09-10 05:56:49.249207: Epoch time: 254.46 s 
+2024-09-10 05:56:51.988348:  
+2024-09-10 05:56:51.989612: Epoch 922 
+2024-09-10 05:56:51.990462: Current learning rate: 0.00101 
+2024-09-10 06:01:09.324709: train_loss -0.7573 
+2024-09-10 06:01:09.336730: val_loss -0.7664 
+2024-09-10 06:01:09.337657: Pseudo dice [0.9691, 0.9632, 0.9843, 0.9582, 0.8899, 0.975, 0.9924] 
+2024-09-10 06:01:09.338460: Epoch time: 257.34 s 
+2024-09-10 06:01:11.419113:  
+2024-09-10 06:01:11.420202: Epoch 923 
+2024-09-10 06:01:11.421014: Current learning rate: 0.001 
+2024-09-10 06:05:23.400592: train_loss -0.7717 
+2024-09-10 06:05:23.402701: val_loss -0.7715 
+2024-09-10 06:05:23.403664: Pseudo dice [0.97, 0.9613, 0.9819, 0.9594, 0.898, 0.9768, 0.9926] 
+2024-09-10 06:05:23.404582: Epoch time: 251.99 s 
+2024-09-10 06:05:23.405368: Yayy! New best EMA pseudo Dice: 0.9589 
+2024-09-10 06:05:29.043216:  
+2024-09-10 06:05:29.044240: Epoch 924 
+2024-09-10 06:05:29.045117: Current learning rate: 0.00098 
+2024-09-10 06:09:42.372036: train_loss -0.7689 
+2024-09-10 06:09:42.374274: val_loss -0.7219 
+2024-09-10 06:09:42.375167: Pseudo dice [0.9691, 0.9605, 0.981, 0.9562, 0.8903, 0.9766, 0.9911] 
+2024-09-10 06:09:42.376091: Epoch time: 253.33 s 
+2024-09-10 06:09:42.376918: Yayy! New best EMA pseudo Dice: 0.9591 
+2024-09-10 06:09:47.828635:  
+2024-09-10 06:09:47.829526: Epoch 925 
+2024-09-10 06:09:47.830235: Current learning rate: 0.00097 
+2024-09-10 06:14:04.575035: train_loss -0.7651 
+2024-09-10 06:14:04.590035: val_loss -0.7415 
+2024-09-10 06:14:04.590837: Pseudo dice [0.9698, 0.9612, 0.9818, 0.9588, 0.8872, 0.9591, 0.9895] 
+2024-09-10 06:14:04.591595: Epoch time: 256.75 s 
+2024-09-10 06:14:07.597708:  
+2024-09-10 06:14:07.598728: Epoch 926 
+2024-09-10 06:14:07.599530: Current learning rate: 0.00096 
+2024-09-10 06:18:15.686060: train_loss -0.7659 
+2024-09-10 06:18:15.699839: val_loss -0.7494 
+2024-09-10 06:18:15.700713: Pseudo dice [0.9689, 0.9606, 0.9807, 0.9468, 0.8666, 0.9762, 0.9904] 
+2024-09-10 06:18:15.701474: Epoch time: 248.09 s 
+2024-09-10 06:18:17.925530:  
+2024-09-10 06:18:17.948065: Epoch 927 
+2024-09-10 06:18:17.956848: Current learning rate: 0.00095 
+2024-09-10 06:22:24.804133: train_loss -0.7678 
+2024-09-10 06:22:24.806863: val_loss -0.753 
+2024-09-10 06:22:24.807850: Pseudo dice [0.9688, 0.9609, 0.981, 0.9559, 0.8955, 0.9752, 0.9922] 
+2024-09-10 06:22:24.822645: Epoch time: 246.88 s 
+2024-09-10 06:22:27.048846:  
+2024-09-10 06:22:27.050052: Epoch 928 
+2024-09-10 06:22:27.050946: Current learning rate: 0.00094 
+2024-09-10 06:26:48.164773: train_loss -0.7473 
+2024-09-10 06:26:48.168431: val_loss -0.7578 
+2024-09-10 06:26:48.169276: Pseudo dice [0.9648, 0.9609, 0.9788, 0.9551, 0.8759, 0.9663, 0.9871] 
+2024-09-10 06:26:48.170074: Epoch time: 261.12 s 
+2024-09-10 06:26:50.286616:  
+2024-09-10 06:26:50.296858: Epoch 929 
+2024-09-10 06:26:50.297937: Current learning rate: 0.00092 
+2024-09-10 06:31:01.881661: train_loss -0.7652 
+2024-09-10 06:31:01.884202: val_loss -0.7412 
+2024-09-10 06:31:01.885150: Pseudo dice [0.9662, 0.9593, 0.9783, 0.9557, 0.8962, 0.9775, 0.9933] 
+2024-09-10 06:31:01.886064: Epoch time: 251.6 s 
+2024-09-10 06:31:04.006404:  
+2024-09-10 06:31:04.007437: Epoch 930 
+2024-09-10 06:31:04.008301: Current learning rate: 0.00091 
+2024-09-10 06:35:26.414431: train_loss -0.7559 
+2024-09-10 06:35:26.415974: val_loss -0.7606 
+2024-09-10 06:35:26.417326: Pseudo dice [0.9667, 0.9637, 0.979, 0.9578, 0.8852, 0.9749, 0.99] 
+2024-09-10 06:35:26.418282: Epoch time: 262.41 s 
+2024-09-10 06:35:30.239909:  
+2024-09-10 06:35:30.241150: Epoch 931 
+2024-09-10 06:35:30.242082: Current learning rate: 0.0009 
+2024-09-10 06:39:53.822426: train_loss -0.7518 
+2024-09-10 06:39:53.828318: val_loss -0.7826 
+2024-09-10 06:39:53.829212: Pseudo dice [0.9674, 0.9581, 0.9808, 0.9544, 0.8937, 0.9768, 0.9926] 
+2024-09-10 06:39:53.829923: Epoch time: 263.59 s 
+2024-09-10 06:39:56.325358:  
+2024-09-10 06:39:56.326773: Epoch 932 
+2024-09-10 06:39:56.327600: Current learning rate: 0.00089 
+2024-09-10 06:44:09.073115: train_loss -0.7784 
+2024-09-10 06:44:09.083470: val_loss -0.7548 
+2024-09-10 06:44:09.084514: Pseudo dice [0.9659, 0.9611, 0.977, 0.9493, 0.8915, 0.9559, 0.9894] 
+2024-09-10 06:44:09.085604: Epoch time: 252.75 s 
+2024-09-10 06:44:11.545546:  
+2024-09-10 06:44:11.561062: Epoch 933 
+2024-09-10 06:44:11.562012: Current learning rate: 0.00088 
+2024-09-10 06:48:20.242607: train_loss -0.7798 
+2024-09-10 06:48:20.244939: val_loss -0.7651 
+2024-09-10 06:48:20.245767: Pseudo dice [0.9698, 0.9627, 0.9817, 0.9593, 0.8961, 0.9773, 0.9919] 
+2024-09-10 06:48:20.246538: Epoch time: 248.7 s 
+2024-09-10 06:48:20.247271: Yayy! New best EMA pseudo Dice: 0.9591 
+2024-09-10 06:48:26.834677:  
+2024-09-10 06:48:26.835728: Epoch 934 
+2024-09-10 06:48:26.836558: Current learning rate: 0.00087 
+2024-09-10 06:52:43.097926: train_loss -0.7688 
+2024-09-10 06:52:43.122013: val_loss -0.7461 
+2024-09-10 06:52:43.122771: Pseudo dice [0.9701, 0.9623, 0.9824, 0.9543, 0.8744, 0.9768, 0.9905] 
+2024-09-10 06:52:43.123448: Epoch time: 256.27 s 
+2024-09-10 06:52:44.963970:  
+2024-09-10 06:52:44.964965: Epoch 935 
+2024-09-10 06:52:44.965757: Current learning rate: 0.00085 
+2024-09-10 06:57:00.275165: train_loss -0.7499 
+2024-09-10 06:57:00.277745: val_loss -0.7745 
+2024-09-10 06:57:00.279010: Pseudo dice [0.9683, 0.9595, 0.9794, 0.9505, 0.9022, 0.9773, 0.9935] 
+2024-09-10 06:57:00.280115: Epoch time: 255.31 s 
+2024-09-10 06:57:00.281114: Yayy! New best EMA pseudo Dice: 0.9593 
+2024-09-10 06:57:06.003128:  
+2024-09-10 06:57:06.004707: Epoch 936 
+2024-09-10 06:57:06.006065: Current learning rate: 0.00084 
+2024-09-10 07:01:05.555847: train_loss -0.7516 
+2024-09-10 07:01:05.575397: val_loss -0.7568 
+2024-09-10 07:01:05.579184: Pseudo dice [0.9689, 0.9616, 0.9807, 0.9549, 0.8845, 0.9719, 0.9911] 
+2024-09-10 07:01:05.585726: Epoch time: 239.56 s 
+2024-09-10 07:01:07.908835:  
+2024-09-10 07:01:07.909919: Epoch 937 
+2024-09-10 07:01:07.910908: Current learning rate: 0.00083 
+2024-09-10 07:05:27.045790: train_loss -0.7705 
+2024-09-10 07:05:27.058914: val_loss -0.7513 
+2024-09-10 07:05:27.059985: Pseudo dice [0.9701, 0.9629, 0.9829, 0.9525, 0.8787, 0.9661, 0.9884] 
+2024-09-10 07:05:27.060909: Epoch time: 259.14 s 
+2024-09-10 07:05:29.499325:  
+2024-09-10 07:05:29.500722: Epoch 938 
+2024-09-10 07:05:29.501672: Current learning rate: 0.00082 
+2024-09-10 07:09:47.254112: train_loss -0.7547 
+2024-09-10 07:09:47.256373: val_loss -0.7513 
+2024-09-10 07:09:47.257279: Pseudo dice [0.9696, 0.9628, 0.9815, 0.9549, 0.865, 0.9773, 0.9887] 
+2024-09-10 07:09:47.258163: Epoch time: 257.76 s 
+2024-09-10 07:09:49.519105:  
+2024-09-10 07:09:49.532917: Epoch 939 
+2024-09-10 07:09:49.534052: Current learning rate: 0.00081 
+2024-09-10 07:14:11.524248: train_loss -0.7669 
+2024-09-10 07:14:11.531827: val_loss -0.772 
+2024-09-10 07:14:11.532851: Pseudo dice [0.9676, 0.9575, 0.9807, 0.9489, 0.8648, 0.9742, 0.9894] 
+2024-09-10 07:14:11.533763: Epoch time: 262.01 s 
+2024-09-10 07:14:13.805543:  
+2024-09-10 07:14:13.806805: Epoch 940 
+2024-09-10 07:14:13.807889: Current learning rate: 0.00079 
+2024-09-10 07:18:28.167822: train_loss -0.7657 
+2024-09-10 07:18:28.186063: val_loss -0.7881 
+2024-09-10 07:18:28.186927: Pseudo dice [0.97, 0.9639, 0.985, 0.9557, 0.8961, 0.9757, 0.9924] 
+2024-09-10 07:18:28.187697: Epoch time: 254.37 s 
+2024-09-10 07:18:30.361286:  
+2024-09-10 07:18:30.362671: Epoch 941 
+2024-09-10 07:18:30.363530: Current learning rate: 0.00078 
+2024-09-10 07:22:53.372204: train_loss -0.7586 
+2024-09-10 07:22:53.374609: val_loss -0.7301 
+2024-09-10 07:22:53.375444: Pseudo dice [0.9676, 0.9608, 0.9773, 0.956, 0.8679, 0.9735, 0.9877] 
+2024-09-10 07:22:53.376293: Epoch time: 263.02 s 
+2024-09-10 07:22:55.545067:  
+2024-09-10 07:22:55.546203: Epoch 942 
+2024-09-10 07:22:55.547205: Current learning rate: 0.00077 
+2024-09-10 07:27:19.859588: train_loss -0.7679 
+2024-09-10 07:27:19.881313: val_loss -0.77 
+2024-09-10 07:27:19.882209: Pseudo dice [0.9693, 0.9609, 0.9798, 0.9527, 0.8922, 0.9661, 0.9922] 
+2024-09-10 07:27:19.883103: Epoch time: 264.32 s 
+2024-09-10 07:27:22.062572:  
+2024-09-10 07:27:22.063637: Epoch 943 
+2024-09-10 07:27:22.064444: Current learning rate: 0.00076 
+2024-09-10 07:31:35.353094: train_loss -0.7494 
+2024-09-10 07:31:35.363405: val_loss -0.7465 
+2024-09-10 07:31:35.364335: Pseudo dice [0.9675, 0.9606, 0.9816, 0.9564, 0.8933, 0.9783, 0.9938] 
+2024-09-10 07:31:35.365239: Epoch time: 253.3 s 
+2024-09-10 07:31:39.000305:  
+2024-09-10 07:31:39.001380: Epoch 944 
+2024-09-10 07:31:39.002299: Current learning rate: 0.00075 
+2024-09-10 07:35:49.533276: train_loss -0.7413 
+2024-09-10 07:35:49.535599: val_loss -0.7577 
+2024-09-10 07:35:49.536375: Pseudo dice [0.9718, 0.9618, 0.9829, 0.9563, 0.9115, 0.9778, 0.9936] 
+2024-09-10 07:35:49.537128: Epoch time: 250.54 s 
+2024-09-10 07:35:49.537812: Yayy! New best EMA pseudo Dice: 0.9596 
+2024-09-10 07:35:55.534897:  
+2024-09-10 07:35:55.536010: Epoch 945 
+2024-09-10 07:35:55.536962: Current learning rate: 0.00074 
+2024-09-10 07:40:22.635794: train_loss -0.7623 
+2024-09-10 07:40:22.651957: val_loss -0.7582 
+2024-09-10 07:40:22.652942: Pseudo dice [0.9643, 0.963, 0.9744, 0.9541, 0.8736, 0.9766, 0.9883] 
+2024-09-10 07:40:22.653785: Epoch time: 267.11 s 
+2024-09-10 07:40:25.089509:  
+2024-09-10 07:40:25.090676: Epoch 946 
+2024-09-10 07:40:25.091514: Current learning rate: 0.00072 
+2024-09-10 07:44:43.533230: train_loss -0.7482 
+2024-09-10 07:44:43.535627: val_loss -0.7654 
+2024-09-10 07:44:43.536483: Pseudo dice [0.9692, 0.963, 0.9805, 0.9531, 0.8938, 0.9773, 0.9915] 
+2024-09-10 07:44:43.537310: Epoch time: 258.45 s 
+2024-09-10 07:44:45.456373:  
+2024-09-10 07:44:45.457400: Epoch 947 
+2024-09-10 07:44:45.458195: Current learning rate: 0.00071 
+2024-09-10 07:49:16.590781: train_loss -0.7474 
+2024-09-10 07:49:16.613976: val_loss -0.7623 
+2024-09-10 07:49:16.614937: Pseudo dice [0.9677, 0.9624, 0.9788, 0.9546, 0.8833, 0.9749, 0.9904] 
+2024-09-10 07:49:16.615774: Epoch time: 271.14 s 
+2024-09-10 07:49:19.099542:  
+2024-09-10 07:49:19.100855: Epoch 948 
+2024-09-10 07:49:19.101755: Current learning rate: 0.0007 
+2024-09-10 07:53:31.166726: train_loss -0.75 
+2024-09-10 07:53:31.172567: val_loss -0.7465 
+2024-09-10 07:53:31.173455: Pseudo dice [0.9703, 0.9605, 0.9824, 0.958, 0.8433, 0.9598, 0.985] 
+2024-09-10 07:53:31.174263: Epoch time: 252.07 s 
+2024-09-10 07:53:33.730836:  
+2024-09-10 07:53:33.732014: Epoch 949 
+2024-09-10 07:53:33.732848: Current learning rate: 0.00069 
+2024-09-10 07:57:57.530195: train_loss -0.7663 
+2024-09-10 07:57:57.549515: val_loss -0.7694 
+2024-09-10 07:57:57.554563: Pseudo dice [0.9684, 0.9612, 0.9806, 0.956, 0.8978, 0.9771, 0.9925] 
+2024-09-10 07:57:57.558441: Epoch time: 263.8 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_10_08_29_27.txt

@@ -0,0 +1,1054 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-10 08:29:37.225371: Using torch.compile... 
+2024-09-10 08:29:48.483300: do_dummy_2d_data_aug: False 
+2024-09-10 08:29:48.489361: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-10 08:29:48.491213: The split file contains 5 splits. 
+2024-09-10 08:29:48.491637: Desired fold for training: 1 
+2024-09-10 08:29:48.492125: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-10 08:30:01.741758: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-10 08:30:06.133267:  
+2024-09-10 08:30:06.137024: Epoch 950 
+2024-09-10 08:30:06.137905: Current learning rate: 0.00067 
+2024-09-10 08:46:48.554314: train_loss -0.7582 
+2024-09-10 08:46:48.555476: val_loss -0.7187 
+2024-09-10 08:46:48.556108: Pseudo dice [0.9692, 0.9629, 0.9795, 0.9599, 0.8714, 0.9772, 0.9887] 
+2024-09-10 08:46:48.556695: Epoch time: 1002.43 s 
+2024-09-10 08:46:50.106956:  
+2024-09-10 08:46:50.107785: Epoch 951 
+2024-09-10 08:46:50.108374: Current learning rate: 0.00066 
+2024-09-10 08:51:26.838150: train_loss -0.7619 
+2024-09-10 08:51:26.856495: val_loss -0.7577 
+2024-09-10 08:51:26.857786: Pseudo dice [0.9677, 0.9615, 0.9796, 0.9502, 0.8786, 0.9762, 0.991] 
+2024-09-10 08:51:26.859135: Epoch time: 276.73 s 
+2024-09-10 08:51:29.367889:  
+2024-09-10 08:51:29.380855: Epoch 952 
+2024-09-10 08:51:29.381869: Current learning rate: 0.00065 
+2024-09-10 08:55:56.799245: train_loss -0.7591 
+2024-09-10 08:55:56.826287: val_loss -0.7486 
+2024-09-10 08:55:56.827275: Pseudo dice [0.9688, 0.9611, 0.9822, 0.9552, 0.8716, 0.9604, 0.9868] 
+2024-09-10 08:55:56.828213: Epoch time: 267.44 s 
+2024-09-10 08:55:59.566823:  
+2024-09-10 08:55:59.567679: Epoch 953 
+2024-09-10 08:55:59.568444: Current learning rate: 0.00064 
+2024-09-10 09:00:26.893699: train_loss -0.7581 
+2024-09-10 09:00:26.895701: val_loss -0.7805 
+2024-09-10 09:00:26.907245: Pseudo dice [0.9675, 0.961, 0.9787, 0.9557, 0.8927, 0.9759, 0.9915] 
+2024-09-10 09:00:26.920708: Epoch time: 267.33 s 
+2024-09-10 09:00:30.431035:  
+2024-09-10 09:00:30.431942: Epoch 954 
+2024-09-10 09:00:30.432680: Current learning rate: 0.00063 
+2024-09-10 09:04:49.909173: train_loss -0.7694 
+2024-09-10 09:04:49.929714: val_loss -0.7576 
+2024-09-10 09:04:49.930436: Pseudo dice [0.9687, 0.9614, 0.9821, 0.9529, 0.8835, 0.9771, 0.9924] 
+2024-09-10 09:04:49.931133: Epoch time: 259.48 s 
+2024-09-10 09:04:53.355830:  
+2024-09-10 09:04:53.368724: Epoch 955 
+2024-09-10 09:04:53.369674: Current learning rate: 0.00061 
+2024-09-10 09:09:30.206495: train_loss -0.7567 
+2024-09-10 09:09:30.234628: val_loss -0.7701 
+2024-09-10 09:09:30.236053: Pseudo dice [0.967, 0.9609, 0.9787, 0.9512, 0.8697, 0.9772, 0.9889] 
+2024-09-10 09:09:30.237138: Epoch time: 276.86 s 
+2024-09-10 09:09:33.451286:  
+2024-09-10 09:09:33.463702: Epoch 956 
+2024-09-10 09:09:33.464659: Current learning rate: 0.0006 
+2024-09-10 09:14:27.922222: train_loss -0.7609 
+2024-09-10 09:14:27.937140: val_loss -0.7529 
+2024-09-10 09:14:27.937896: Pseudo dice [0.97, 0.9616, 0.9822, 0.9553, 0.8968, 0.9664, 0.9913] 
+2024-09-10 09:14:27.938678: Epoch time: 294.48 s 
+2024-09-10 09:14:31.208779:  
+2024-09-10 09:14:31.220621: Epoch 957 
+2024-09-10 09:14:31.221415: Current learning rate: 0.00059 
+2024-09-10 09:19:03.239066: train_loss -0.7488 
+2024-09-10 09:19:03.254275: val_loss -0.7532 
+2024-09-10 09:19:03.255178: Pseudo dice [0.9697, 0.9608, 0.9829, 0.9585, 0.8962, 0.9511, 0.9873] 
+2024-09-10 09:19:03.256019: Epoch time: 272.04 s 
+2024-09-10 09:19:05.895707:  
+2024-09-10 09:19:05.912103: Epoch 958 
+2024-09-10 09:19:05.913097: Current learning rate: 0.00058 
+2024-09-10 09:23:49.792643: train_loss -0.7706 
+2024-09-10 09:23:49.793969: val_loss -0.7614 
+2024-09-10 09:23:49.794815: Pseudo dice [0.9679, 0.9612, 0.9787, 0.954, 0.886, 0.9758, 0.9916] 
+2024-09-10 09:23:49.795621: Epoch time: 283.9 s 
+2024-09-10 09:23:53.243169:  
+2024-09-10 09:23:53.256748: Epoch 959 
+2024-09-10 09:23:53.257547: Current learning rate: 0.00056 
+2024-09-10 09:28:33.890625: train_loss -0.7648 
+2024-09-10 09:28:33.905273: val_loss -0.7623 
+2024-09-10 09:28:33.906444: Pseudo dice [0.9675, 0.9592, 0.9801, 0.949, 0.8513, 0.9766, 0.9884] 
+2024-09-10 09:28:33.907589: Epoch time: 280.65 s 
+2024-09-10 09:28:36.284932:  
+2024-09-10 09:28:36.297088: Epoch 960 
+2024-09-10 09:28:36.298202: Current learning rate: 0.00055 
+2024-09-10 09:33:16.327193: train_loss -0.7613 
+2024-09-10 09:33:16.331599: val_loss -0.7809 
+2024-09-10 09:33:16.336712: Pseudo dice [0.9696, 0.9601, 0.9818, 0.956, 0.9057, 0.9546, 0.9883] 
+2024-09-10 09:33:16.337470: Epoch time: 280.05 s 
+2024-09-10 09:33:18.957974:  
+2024-09-10 09:33:18.958792: Epoch 961 
+2024-09-10 09:33:18.959445: Current learning rate: 0.00054 
+2024-09-10 09:38:10.198527: train_loss -0.7683 
+2024-09-10 09:38:10.213464: val_loss -0.7777 
+2024-09-10 09:38:10.214437: Pseudo dice [0.9681, 0.9581, 0.983, 0.9561, 0.8896, 0.9637, 0.9891] 
+2024-09-10 09:38:10.215422: Epoch time: 291.24 s 
+2024-09-10 09:38:12.847270:  
+2024-09-10 09:38:12.864103: Epoch 962 
+2024-09-10 09:38:12.865067: Current learning rate: 0.00053 
+2024-09-10 09:42:48.379891: train_loss -0.7731 
+2024-09-10 09:42:48.382085: val_loss -0.7822 
+2024-09-10 09:42:48.382762: Pseudo dice [0.9708, 0.9625, 0.9845, 0.9579, 0.9003, 0.966, 0.9906] 
+2024-09-10 09:42:48.383580: Epoch time: 275.54 s 
+2024-09-10 09:42:52.009767:  
+2024-09-10 09:42:52.024778: Epoch 963 
+2024-09-10 09:42:52.025777: Current learning rate: 0.00051 
+2024-09-10 09:47:32.489001: train_loss -0.7714 
+2024-09-10 09:47:32.505862: val_loss -0.7731 
+2024-09-10 09:47:32.506644: Pseudo dice [0.9704, 0.9628, 0.9832, 0.9583, 0.884, 0.9654, 0.9883] 
+2024-09-10 09:47:32.507293: Epoch time: 280.48 s 
+2024-09-10 09:47:36.159363:  
+2024-09-10 09:47:36.172861: Epoch 964 
+2024-09-10 09:47:36.173662: Current learning rate: 0.0005 
+2024-09-10 09:51:48.444608: train_loss -0.773 
+2024-09-10 09:51:48.451307: val_loss -0.7757 
+2024-09-10 09:51:48.452085: Pseudo dice [0.9676, 0.9612, 0.9784, 0.9569, 0.8811, 0.9773, 0.99] 
+2024-09-10 09:51:48.452927: Epoch time: 252.29 s 
+2024-09-10 09:51:51.128967:  
+2024-09-10 09:51:51.130070: Epoch 965 
+2024-09-10 09:51:51.130783: Current learning rate: 0.00049 
+2024-09-10 09:56:02.366265: train_loss -0.7577 
+2024-09-10 09:56:02.381255: val_loss -0.7757 
+2024-09-10 09:56:02.382028: Pseudo dice [0.9699, 0.9626, 0.9825, 0.9571, 0.8887, 0.9774, 0.9916] 
+2024-09-10 09:56:02.382985: Epoch time: 251.24 s 
+2024-09-10 09:56:05.958422:  
+2024-09-10 09:56:05.959858: Epoch 966 
+2024-09-10 09:56:05.961766: Current learning rate: 0.00048 
+2024-09-10 10:00:29.981308: train_loss -0.7582 
+2024-09-10 10:00:29.989092: val_loss -0.7718 
+2024-09-10 10:00:29.990085: Pseudo dice [0.9678, 0.9624, 0.9781, 0.9585, 0.8806, 0.9769, 0.9895] 
+2024-09-10 10:00:29.991668: Epoch time: 264.03 s 
+2024-09-10 10:00:33.916290:  
+2024-09-10 10:00:33.928711: Epoch 967 
+2024-09-10 10:00:33.929532: Current learning rate: 0.00046 
+2024-09-10 10:05:13.090747: train_loss -0.7661 
+2024-09-10 10:05:13.108236: val_loss -0.7717 
+2024-09-10 10:05:13.109250: Pseudo dice [0.969, 0.9606, 0.9796, 0.9566, 0.8961, 0.9772, 0.9926] 
+2024-09-10 10:05:13.109971: Epoch time: 279.18 s 
+2024-09-10 10:05:15.813483:  
+2024-09-10 10:05:15.814380: Epoch 968 
+2024-09-10 10:05:15.814975: Current learning rate: 0.00045 
+2024-09-10 10:09:38.244115: train_loss -0.7761 
+2024-09-10 10:09:38.245542: val_loss -0.7516 
+2024-09-10 10:09:38.246348: Pseudo dice [0.9673, 0.9613, 0.9795, 0.9527, 0.8785, 0.9633, 0.9862] 
+2024-09-10 10:09:38.247142: Epoch time: 262.44 s 
+2024-09-10 10:09:41.255548:  
+2024-09-10 10:09:41.268698: Epoch 969 
+2024-09-10 10:09:41.269500: Current learning rate: 0.00044 
+2024-09-10 10:14:09.710062: train_loss -0.7756 
+2024-09-10 10:14:09.712465: val_loss -0.7452 
+2024-09-10 10:14:09.713173: Pseudo dice [0.9683, 0.9604, 0.9799, 0.9556, 0.8861, 0.9758, 0.9909] 
+2024-09-10 10:14:09.713756: Epoch time: 268.46 s 
+2024-09-10 10:14:13.009937:  
+2024-09-10 10:14:13.010835: Epoch 970 
+2024-09-10 10:14:13.011470: Current learning rate: 0.00043 
+2024-09-10 10:18:33.108193: train_loss -0.7591 
+2024-09-10 10:18:33.123045: val_loss -0.7443 
+2024-09-10 10:18:33.124199: Pseudo dice [0.9675, 0.9614, 0.9774, 0.9508, 0.877, 0.976, 0.9893] 
+2024-09-10 10:18:33.125107: Epoch time: 260.1 s 
+2024-09-10 10:18:35.943322:  
+2024-09-10 10:18:35.944194: Epoch 971 
+2024-09-10 10:18:35.944920: Current learning rate: 0.00041 
+2024-09-10 10:22:58.904608: train_loss -0.7649 
+2024-09-10 10:22:58.906815: val_loss -0.778 
+2024-09-10 10:22:58.907553: Pseudo dice [0.9658, 0.9601, 0.9781, 0.9515, 0.8683, 0.9742, 0.9884] 
+2024-09-10 10:22:58.908133: Epoch time: 262.97 s 
+2024-09-10 10:23:02.729292:  
+2024-09-10 10:23:02.730444: Epoch 972 
+2024-09-10 10:23:02.731252: Current learning rate: 0.0004 
+2024-09-10 10:27:41.171043: train_loss -0.7645 
+2024-09-10 10:27:41.173527: val_loss -0.7401 
+2024-09-10 10:27:41.174312: Pseudo dice [0.9695, 0.9615, 0.9814, 0.9555, 0.8758, 0.9645, 0.9882] 
+2024-09-10 10:27:41.174980: Epoch time: 278.45 s 
+2024-09-10 10:27:44.005050:  
+2024-09-10 10:27:44.005888: Epoch 973 
+2024-09-10 10:27:44.006670: Current learning rate: 0.00039 
+2024-09-10 10:32:12.216469: train_loss -0.7757 
+2024-09-10 10:32:12.231104: val_loss -0.7573 
+2024-09-10 10:32:12.231826: Pseudo dice [0.9681, 0.9592, 0.9807, 0.9544, 0.8845, 0.9736, 0.9913] 
+2024-09-10 10:32:12.233620: Epoch time: 268.22 s 
+2024-09-10 10:32:15.022367:  
+2024-09-10 10:32:15.032091: Epoch 974 
+2024-09-10 10:32:15.033014: Current learning rate: 0.00037 
+2024-09-10 10:36:46.559494: train_loss -0.7649 
+2024-09-10 10:36:46.573973: val_loss -0.8081 
+2024-09-10 10:36:46.574748: Pseudo dice [0.9695, 0.959, 0.9814, 0.9571, 0.8996, 0.977, 0.9931] 
+2024-09-10 10:36:46.575504: Epoch time: 271.54 s 
+2024-09-10 10:36:49.973804:  
+2024-09-10 10:36:49.984785: Epoch 975 
+2024-09-10 10:36:49.985707: Current learning rate: 0.00036 
+2024-09-10 10:41:20.572410: train_loss -0.7442 
+2024-09-10 10:41:20.588845: val_loss -0.7394 
+2024-09-10 10:41:20.589780: Pseudo dice [0.9701, 0.9618, 0.9821, 0.9571, 0.8697, 0.9632, 0.9857] 
+2024-09-10 10:41:20.590553: Epoch time: 270.6 s 
+2024-09-10 10:41:23.969845:  
+2024-09-10 10:41:23.970845: Epoch 976 
+2024-09-10 10:41:23.971596: Current learning rate: 0.00035 
+2024-09-10 10:46:17.799031: train_loss -0.7681 
+2024-09-10 10:46:17.801483: val_loss -0.8065 
+2024-09-10 10:46:17.802342: Pseudo dice [0.971, 0.9614, 0.9825, 0.9567, 0.8902, 0.9759, 0.991] 
+2024-09-10 10:46:17.803077: Epoch time: 293.83 s 
+2024-09-10 10:46:20.922901:  
+2024-09-10 10:46:20.923695: Epoch 977 
+2024-09-10 10:46:20.940289: Current learning rate: 0.00034 
+2024-09-10 10:50:57.227098: train_loss -0.783 
+2024-09-10 10:50:57.231740: val_loss -0.7918 
+2024-09-10 10:50:57.234128: Pseudo dice [0.9705, 0.9631, 0.9828, 0.9614, 0.9113, 0.9783, 0.9931] 
+2024-09-10 10:50:57.241935: Epoch time: 276.31 s 
+2024-09-10 10:51:00.891597:  
+2024-09-10 10:51:00.904808: Epoch 978 
+2024-09-10 10:51:00.905723: Current learning rate: 0.00032 
+2024-09-10 10:55:37.234911: train_loss -0.7787 
+2024-09-10 10:55:37.243323: val_loss -0.7591 
+2024-09-10 10:55:37.244189: Pseudo dice [0.9679, 0.9615, 0.979, 0.9566, 0.9043, 0.978, 0.9917] 
+2024-09-10 10:55:37.245078: Epoch time: 276.35 s 
+2024-09-10 10:55:37.245728: Yayy! New best EMA pseudo Dice: 0.9597 
+2024-09-10 10:55:43.240841:  
+2024-09-10 10:55:43.241717: Epoch 979 
+2024-09-10 10:55:43.242448: Current learning rate: 0.00031 
+2024-09-10 11:00:24.032337: train_loss -0.7673 
+2024-09-10 11:00:24.052553: val_loss -0.7539 
+2024-09-10 11:00:24.053331: Pseudo dice [0.9693, 0.9634, 0.9819, 0.9597, 0.8914, 0.9776, 0.9912] 
+2024-09-10 11:00:24.054076: Epoch time: 280.8 s 
+2024-09-10 11:00:24.054729: Yayy! New best EMA pseudo Dice: 0.96 
+2024-09-10 11:00:29.866855:  
+2024-09-10 11:00:29.867837: Epoch 980 
+2024-09-10 11:00:29.868663: Current learning rate: 0.0003 
+2024-09-10 11:05:08.868864: train_loss -0.7773 
+2024-09-10 11:05:08.885309: val_loss -0.7752 
+2024-09-10 11:05:08.886329: Pseudo dice [0.9686, 0.9606, 0.9807, 0.9574, 0.8975, 0.9771, 0.9931] 
+2024-09-10 11:05:08.887120: Epoch time: 279.01 s 
+2024-09-10 11:05:08.887903: Yayy! New best EMA pseudo Dice: 0.9602 
+2024-09-10 11:05:15.459151:  
+2024-09-10 11:05:15.460121: Epoch 981 
+2024-09-10 11:05:15.460822: Current learning rate: 0.00028 
+2024-09-10 11:09:50.410739: train_loss -0.7734 
+2024-09-10 11:09:50.437191: val_loss -0.7799 
+2024-09-10 11:09:50.438245: Pseudo dice [0.9676, 0.9601, 0.9792, 0.9563, 0.8946, 0.9735, 0.9906] 
+2024-09-10 11:09:50.439071: Epoch time: 274.96 s 
+2024-09-10 11:09:50.439735: Yayy! New best EMA pseudo Dice: 0.9602 
+2024-09-10 11:09:56.282006:  
+2024-09-10 11:09:56.282942: Epoch 982 
+2024-09-10 11:09:56.283646: Current learning rate: 0.00027 
+2024-09-10 11:14:26.459307: train_loss -0.7618 
+2024-09-10 11:14:26.480921: val_loss -0.7747 
+2024-09-10 11:14:26.481755: Pseudo dice [0.9671, 0.9573, 0.9808, 0.9556, 0.9006, 0.9661, 0.9908] 
+2024-09-10 11:14:26.482472: Epoch time: 270.18 s 
+2024-09-10 11:14:30.617548:  
+2024-09-10 11:14:30.618746: Epoch 983 
+2024-09-10 11:14:30.619467: Current learning rate: 0.00026 
+2024-09-10 11:19:00.162123: train_loss -0.7653 
+2024-09-10 11:19:00.175327: val_loss -0.7662 
+2024-09-10 11:19:00.176429: Pseudo dice [0.97, 0.9646, 0.9832, 0.9548, 0.8669, 0.9764, 0.9881] 
+2024-09-10 11:19:00.177421: Epoch time: 269.55 s 
+2024-09-10 11:19:03.405085:  
+2024-09-10 11:19:03.405988: Epoch 984 
+2024-09-10 11:19:03.406575: Current learning rate: 0.00024 
+2024-09-10 11:23:34.307810: train_loss -0.7699 
+2024-09-10 11:23:34.317140: val_loss -0.7769 
+2024-09-10 11:23:34.318020: Pseudo dice [0.9699, 0.9632, 0.9816, 0.9556, 0.8954, 0.9618, 0.9895] 
+2024-09-10 11:23:34.319224: Epoch time: 270.91 s 
+2024-09-10 11:23:37.461198:  
+2024-09-10 11:23:37.462309: Epoch 985 
+2024-09-10 11:23:37.463021: Current learning rate: 0.00023 
+2024-09-10 11:27:58.907676: train_loss -0.7662 
+2024-09-10 11:27:58.922392: val_loss -0.7631 
+2024-09-10 11:27:58.923414: Pseudo dice [0.9721, 0.9647, 0.984, 0.9616, 0.9014, 0.9646, 0.9906] 
+2024-09-10 11:27:58.924388: Epoch time: 261.45 s 
+2024-09-10 11:28:02.344844:  
+2024-09-10 11:28:02.345956: Epoch 986 
+2024-09-10 11:28:02.346606: Current learning rate: 0.00021 
+2024-09-10 11:32:29.299577: train_loss -0.7641 
+2024-09-10 11:32:29.313340: val_loss -0.7897 
+2024-09-10 11:32:29.314418: Pseudo dice [0.9711, 0.9622, 0.9839, 0.9591, 0.8976, 0.9776, 0.9928] 
+2024-09-10 11:32:29.315289: Epoch time: 266.96 s 
+2024-09-10 11:32:29.315901: Yayy! New best EMA pseudo Dice: 0.9605 
+2024-09-10 11:32:36.205130:  
+2024-09-10 11:32:36.206204: Epoch 987 
+2024-09-10 11:32:36.206988: Current learning rate: 0.0002 
+2024-09-10 11:37:02.719902: train_loss -0.7641 
+2024-09-10 11:37:02.721562: val_loss -0.7487 
+2024-09-10 11:37:02.722271: Pseudo dice [0.9694, 0.9624, 0.9806, 0.9627, 0.8954, 0.9759, 0.9903] 
+2024-09-10 11:37:02.722961: Epoch time: 266.52 s 
+2024-09-10 11:37:02.723520: Yayy! New best EMA pseudo Dice: 0.9607 
+2024-09-10 11:37:09.465965:  
+2024-09-10 11:37:09.466956: Epoch 988 
+2024-09-10 11:37:09.467606: Current learning rate: 0.00019 
+2024-09-10 11:41:46.652526: train_loss -0.7908 
+2024-09-10 11:41:46.658344: val_loss -0.7825 
+2024-09-10 11:41:46.659273: Pseudo dice [0.9699, 0.9626, 0.9821, 0.9588, 0.906, 0.9624, 0.991] 
+2024-09-10 11:41:46.663346: Epoch time: 277.19 s 
+2024-09-10 11:41:46.664181: Yayy! New best EMA pseudo Dice: 0.9608 
+2024-09-10 11:41:53.104045:  
+2024-09-10 11:41:53.104858: Epoch 989 
+2024-09-10 11:41:53.105572: Current learning rate: 0.00017 
+2024-09-10 11:46:13.609880: train_loss -0.7607 
+2024-09-10 11:46:13.632907: val_loss -0.8003 
+2024-09-10 11:46:13.633680: Pseudo dice [0.97, 0.9616, 0.9833, 0.9511, 0.8972, 0.9769, 0.9931] 
+2024-09-10 11:46:13.634456: Epoch time: 260.51 s 
+2024-09-10 11:46:13.635124: Yayy! New best EMA pseudo Dice: 0.9609 
+2024-09-10 11:46:19.664327:  
+2024-09-10 11:46:19.665242: Epoch 990 
+2024-09-10 11:46:19.665925: Current learning rate: 0.00016 
+2024-09-10 11:50:30.125223: train_loss -0.787 
+2024-09-10 11:50:30.127539: val_loss -0.8119 
+2024-09-10 11:50:30.128190: Pseudo dice [0.9711, 0.9631, 0.9836, 0.958, 0.8901, 0.9772, 0.9911] 
+2024-09-10 11:50:30.129002: Epoch time: 250.46 s 
+2024-09-10 11:50:30.129649: Yayy! New best EMA pseudo Dice: 0.961 
+2024-09-10 11:50:36.326822:  
+2024-09-10 11:50:36.327598: Epoch 991 
+2024-09-10 11:50:36.328182: Current learning rate: 0.00014 
+2024-09-10 11:55:07.458744: train_loss -0.7678 
+2024-09-10 11:55:07.460057: val_loss -0.7593 
+2024-09-10 11:55:07.460861: Pseudo dice [0.9689, 0.9623, 0.981, 0.96, 0.8949, 0.9776, 0.9919] 
+2024-09-10 11:55:07.461720: Epoch time: 271.14 s 
+2024-09-10 11:55:07.462679: Yayy! New best EMA pseudo Dice: 0.9611 
+2024-09-10 11:55:13.424548:  
+2024-09-10 11:55:13.425507: Epoch 992 
+2024-09-10 11:55:13.426335: Current learning rate: 0.00013 
+2024-09-10 11:59:39.943476: train_loss -0.755 
+2024-09-10 11:59:39.958046: val_loss -0.7773 
+2024-09-10 11:59:39.958901: Pseudo dice [0.9645, 0.9588, 0.9785, 0.9566, 0.8898, 0.9762, 0.992] 
+2024-09-10 11:59:39.959548: Epoch time: 266.52 s 
+2024-09-10 11:59:42.556968:  
+2024-09-10 11:59:42.557827: Epoch 993 
+2024-09-10 11:59:42.558452: Current learning rate: 0.00011 
+2024-09-10 12:03:57.020513: train_loss -0.7582 
+2024-09-10 12:03:57.022946: val_loss -0.7654 
+2024-09-10 12:03:57.023671: Pseudo dice [0.9684, 0.9609, 0.9814, 0.9603, 0.9003, 0.9772, 0.9924] 
+2024-09-10 12:03:57.024410: Epoch time: 254.47 s 
+2024-09-10 12:03:57.025147: Yayy! New best EMA pseudo Dice: 0.9612 
+2024-09-10 12:04:02.861754:  
+2024-09-10 12:04:02.862796: Epoch 994 
+2024-09-10 12:04:02.863601: Current learning rate: 0.0001 
+2024-09-10 12:08:31.567364: train_loss -0.7725 
+2024-09-10 12:08:31.571005: val_loss -0.7757 
+2024-09-10 12:08:31.571710: Pseudo dice [0.9691, 0.9609, 0.9815, 0.9542, 0.8984, 0.9662, 0.9909] 
+2024-09-10 12:08:31.572357: Epoch time: 268.71 s 
+2024-09-10 12:08:33.602363:  
+2024-09-10 12:08:33.616684: Epoch 995 
+2024-09-10 12:08:33.617510: Current learning rate: 8e-05 
+2024-09-10 12:13:07.659117: train_loss -0.7742 
+2024-09-10 12:13:07.681635: val_loss -0.7667 
+2024-09-10 12:13:07.682946: Pseudo dice [0.9726, 0.9646, 0.9859, 0.9611, 0.9004, 0.9763, 0.9912] 
+2024-09-10 12:13:07.684081: Epoch time: 274.06 s 
+2024-09-10 12:13:07.685158: Yayy! New best EMA pseudo Dice: 0.9614 
+2024-09-10 12:13:13.644514:  
+2024-09-10 12:13:13.645438: Epoch 996 
+2024-09-10 12:13:13.646118: Current learning rate: 7e-05 
+2024-09-10 12:17:38.868143: train_loss -0.7602 
+2024-09-10 12:17:38.885116: val_loss -0.7634 
+2024-09-10 12:17:38.885849: Pseudo dice [0.9697, 0.9636, 0.9822, 0.9593, 0.9072, 0.9768, 0.993] 
+2024-09-10 12:17:38.886549: Epoch time: 265.23 s 
+2024-09-10 12:17:38.887154: Yayy! New best EMA pseudo Dice: 0.9617 
+2024-09-10 12:17:44.499961:  
+2024-09-10 12:17:44.500835: Epoch 997 
+2024-09-10 12:17:44.501553: Current learning rate: 5e-05 
+2024-09-10 12:22:22.249170: train_loss -0.7723 
+2024-09-10 12:22:22.265131: val_loss -0.7964 
+2024-09-10 12:22:22.266015: Pseudo dice [0.9709, 0.962, 0.9813, 0.9509, 0.8992, 0.9718, 0.9911] 
+2024-09-10 12:22:22.266733: Epoch time: 277.75 s 
+2024-09-10 12:22:26.682065:  
+2024-09-10 12:22:26.683177: Epoch 998 
+2024-09-10 12:22:26.683873: Current learning rate: 4e-05 
+2024-09-10 12:26:49.741565: train_loss -0.7812 
+2024-09-10 12:26:49.757162: val_loss -0.8126 
+2024-09-10 12:26:49.758154: Pseudo dice [0.969, 0.959, 0.9837, 0.9585, 0.9024, 0.9782, 0.9935] 
+2024-09-10 12:26:49.758948: Epoch time: 263.06 s 
+2024-09-10 12:26:49.759586: Yayy! New best EMA pseudo Dice: 0.9619 
+2024-09-10 12:26:56.457734:  
+2024-09-10 12:26:56.464732: Epoch 999 
+2024-09-10 12:26:56.465559: Current learning rate: 2e-05 
+2024-09-10 12:31:29.140393: train_loss -0.7407 
+2024-09-10 12:31:29.148701: val_loss -0.7112 
+2024-09-10 12:31:29.149606: Pseudo dice [0.9708, 0.9623, 0.9841, 0.9562, 0.8481, 0.9364, 0.9781] 
+2024-09-10 12:31:29.150458: Epoch time: 272.69 s 
+2024-09-10 12:31:36.262199: Training done. 
+2024-09-10 12:31:36.296587: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-10 12:31:36.299175: The split file contains 5 splits. 
+2024-09-10 12:31:36.299687: Desired fold for training: 1 
+2024-09-10 12:31:36.300214: This split has 1325 training and 331 validation cases. 
+2024-09-10 12:31:36.309337: predicting saros-002 
+2024-09-10 12:31:36.858610: saros-002, shape torch.Size([1, 323, 281, 281]), rank 0 
+2024-09-10 12:33:53.586158: predicting saros-005 
+2024-09-10 12:33:53.991145: saros-005, shape torch.Size([1, 253, 240, 240]), rank 0 
+2024-09-10 12:33:57.678060: predicting saros-007 
+2024-09-10 12:33:58.129935: saros-007, shape torch.Size([1, 323, 227, 227]), rank 0 
+2024-09-10 12:34:03.605711: predicting saros-010 
+2024-09-10 12:34:03.937168: saros-010, shape torch.Size([1, 123, 323, 323]), rank 0 
+2024-09-10 12:34:08.110714: predicting saros-025 
+2024-09-10 12:34:08.640528: saros-025, shape torch.Size([1, 320, 289, 289]), rank 0 
+2024-09-10 12:34:20.941646: predicting saros-028 
+2024-09-10 12:34:21.669165: saros-028, shape torch.Size([1, 297, 309, 309]), rank 0 
+2024-09-10 12:34:33.983100: predicting saros-029 
+2024-09-10 12:34:34.425101: saros-029, shape torch.Size([1, 293, 230, 230]), rank 0 
+2024-09-10 12:34:39.946807: predicting saros-054 
+2024-09-10 12:34:40.269359: saros-054, shape torch.Size([1, 140, 321, 321]), rank 0 
+2024-09-10 12:34:44.475962: predicting saros-055 
+2024-09-10 12:34:45.020467: saros-055, shape torch.Size([1, 300, 267, 267]), rank 0 
+2024-09-10 12:34:50.566189: predicting saros-069 
+2024-09-10 12:34:50.935525: saros-069, shape torch.Size([1, 307, 236, 236]), rank 0 
+2024-09-10 12:34:56.451090: predicting saros-077 
+2024-09-10 12:34:56.748878: saros-077, shape torch.Size([1, 167, 267, 267]), rank 0 
+2024-09-10 12:34:58.690203: predicting saros-079 
+2024-09-10 12:34:59.344941: saros-079, shape torch.Size([1, 313, 300, 300]), rank 0 
+2024-09-10 12:35:11.644070: predicting saros-081 
+2024-09-10 12:35:12.157771: saros-081, shape torch.Size([1, 227, 307, 307]), rank 0 
+2024-09-10 12:35:20.410964: predicting saros-082 
+2024-09-10 12:35:20.709146: saros-082, shape torch.Size([1, 180, 253, 253]), rank 0 
+2024-09-10 12:35:22.632583: predicting saros-084 
+2024-09-10 12:35:23.362446: saros-084, shape torch.Size([1, 343, 298, 298]), rank 0 
+2024-09-10 12:35:35.669823: predicting saros-090 
+2024-09-10 12:35:36.361255: saros-090, shape torch.Size([1, 353, 301, 301]), rank 0 
+2024-09-10 12:35:48.707488: predicting saros-097 
+2024-09-10 12:35:49.095124: saros-097, shape torch.Size([1, 143, 333, 333]), rank 0 
+2024-09-10 12:35:53.317142: predicting saros-101 
+2024-09-10 12:35:53.847534: saros-101, shape torch.Size([1, 313, 253, 253]), rank 0 
+2024-09-10 12:35:59.392470: predicting saros-106 
+2024-09-10 12:36:00.057401: saros-106, shape torch.Size([1, 307, 305, 305]), rank 0 
+2024-09-10 12:36:12.373046: predicting saros-109 
+2024-09-10 12:36:12.894006: saros-109, shape torch.Size([1, 200, 331, 331]), rank 0 
+2024-09-10 12:36:21.151391: predicting saros-118 
+2024-09-10 12:36:21.377269: saros-118, shape torch.Size([1, 207, 240, 240]), rank 0 
+2024-09-10 12:36:25.120202: predicting saros-131 
+2024-09-10 12:36:25.534675: saros-131, shape torch.Size([1, 200, 293, 293]), rank 0 
+2024-09-10 12:36:33.753484: predicting saros-133 
+2024-09-10 12:36:33.955783: saros-133, shape torch.Size([1, 153, 240, 240]), rank 0 
+2024-09-10 12:36:35.846687: predicting saros-135 
+2024-09-10 12:36:36.578053: saros-135, shape torch.Size([1, 330, 333, 333]), rank 0 
+2024-09-10 12:36:48.989578: predicting saros-148 
+2024-09-10 12:36:49.224592: saros-148, shape torch.Size([1, 140, 267, 267]), rank 0 
+2024-09-10 12:36:51.158711: predicting saros-155 
+2024-09-10 12:36:51.763847: saros-155, shape torch.Size([1, 313, 287, 287]), rank 0 
+2024-09-10 12:36:57.369071: predicting saros-158 
+2024-09-10 12:36:57.796457: saros-158, shape torch.Size([1, 273, 253, 253]), rank 0 
+2024-09-10 12:37:01.527429: predicting saros-164 
+2024-09-10 12:37:01.905950: saros-164, shape torch.Size([1, 173, 313, 313]), rank 0 
+2024-09-10 12:37:06.121175: predicting saros-165 
+2024-09-10 12:37:06.387272: saros-165, shape torch.Size([1, 167, 240, 240]), rank 0 
+2024-09-10 12:37:08.332984: predicting saros-172 
+2024-09-10 12:37:08.536488: saros-172, shape torch.Size([1, 147, 237, 237]), rank 0 
+2024-09-10 12:37:10.424962: predicting saros-173 
+2024-09-10 12:37:10.683406: saros-173, shape torch.Size([1, 133, 320, 320]), rank 0 
+2024-09-10 12:37:14.879227: predicting saros-181 
+2024-09-10 12:37:15.431044: saros-181, shape torch.Size([1, 297, 302, 302]), rank 0 
+2024-09-10 12:37:27.753890: predicting saros-185 
+2024-09-10 12:37:28.114309: saros-185, shape torch.Size([1, 143, 333, 333]), rank 0 
+2024-09-10 12:37:32.366340: predicting saros-193 
+2024-09-10 12:37:32.758316: saros-193, shape torch.Size([1, 147, 307, 307]), rank 0 
+2024-09-10 12:37:36.974803: predicting saros-199 
+2024-09-10 12:37:37.341293: saros-199, shape torch.Size([1, 257, 243, 243]), rank 0 
+2024-09-10 12:37:41.068396: predicting saros-212 
+2024-09-10 12:37:41.762642: saros-212, shape torch.Size([1, 327, 329, 329]), rank 0 
+2024-09-10 12:37:54.094620: predicting saros-220 
+2024-09-10 12:37:54.304840: saros-220, shape torch.Size([1, 180, 240, 240]), rank 0 
+2024-09-10 12:37:56.243925: predicting saros-234 
+2024-09-10 12:37:56.584596: saros-234, shape torch.Size([1, 177, 293, 293]), rank 0 
+2024-09-10 12:38:00.757508: predicting saros-244 
+2024-09-10 12:38:01.115180: saros-244, shape torch.Size([1, 150, 320, 320]), rank 0 
+2024-09-10 12:38:05.326512: predicting saros-250 
+2024-09-10 12:38:05.679086: saros-250, shape torch.Size([1, 157, 333, 333]), rank 0 
+2024-09-10 12:38:09.872819: predicting saros-254 
+2024-09-10 12:38:10.350500: saros-254, shape torch.Size([1, 330, 253, 253]), rank 0 
+2024-09-10 12:38:15.909896: predicting saros-259 
+2024-09-10 12:38:16.190290: saros-259, shape torch.Size([1, 130, 307, 307]), rank 0 
+2024-09-10 12:38:20.390612: predicting saros-264 
+2024-09-10 12:38:21.082508: saros-264, shape torch.Size([1, 323, 297, 297]), rank 0 
+2024-09-10 12:38:33.426380: predicting saros-266 
+2024-09-10 12:38:33.656855: saros-266, shape torch.Size([1, 163, 253, 253]), rank 0 
+2024-09-10 12:38:35.596403: predicting saros-275 
+2024-09-10 12:38:36.320090: saros-275, shape torch.Size([1, 343, 325, 325]), rank 0 
+2024-09-10 12:38:48.638834: predicting saros-281 
+2024-09-10 12:38:49.436601: saros-281, shape torch.Size([1, 393, 300, 300]), rank 0 
+2024-09-10 12:39:05.833340: predicting saros-286 
+2024-09-10 12:39:06.040999: saros-286, shape torch.Size([1, 137, 267, 267]), rank 0 
+2024-09-10 12:39:07.983869: predicting saros-290 
+2024-09-10 12:39:08.304111: saros-290, shape torch.Size([1, 183, 276, 276]), rank 0 
+2024-09-10 12:39:10.227714: predicting saros-292 
+2024-09-10 12:39:10.492966: saros-292, shape torch.Size([1, 180, 253, 253]), rank 0 
+2024-09-10 12:39:12.439424: predicting saros-296 
+2024-09-10 12:39:12.872450: saros-296, shape torch.Size([1, 323, 253, 253]), rank 0 
+2024-09-10 12:39:18.399213: predicting saros-300 
+2024-09-10 12:39:18.660909: saros-300, shape torch.Size([1, 203, 217, 217]), rank 0 
+2024-09-10 12:39:22.325221: predicting saros-306 
+2024-09-10 12:39:22.557366: saros-306, shape torch.Size([1, 180, 184, 184]), rank 0 
+2024-09-10 12:39:23.124089: predicting saros-312 
+2024-09-10 12:39:23.319345: saros-312, shape torch.Size([1, 200, 180, 180]), rank 0 
+2024-09-10 12:40:47.406436: predicting saros-316 
+2024-09-10 12:40:47.618143: saros-316, shape torch.Size([1, 220, 221, 221]), rank 0 
+2024-09-10 12:40:51.270763: predicting saros-319 
+2024-09-10 12:40:51.515487: saros-319, shape torch.Size([1, 193, 221, 221]), rank 0 
+2024-09-10 12:40:55.161184: predicting saros-321 
+2024-09-10 12:40:55.406496: saros-321, shape torch.Size([1, 223, 225, 225]), rank 0 
+2024-09-10 12:40:59.110018: predicting saros-323 
+2024-09-10 12:40:59.330532: saros-323, shape torch.Size([1, 187, 195, 195]), rank 0 
+2024-09-10 12:41:01.220541: predicting saros-334 
+2024-09-10 12:41:01.641894: saros-334, shape torch.Size([1, 247, 261, 261]), rank 0 
+2024-09-10 12:41:05.362823: predicting saros-351 
+2024-09-10 12:41:05.591122: saros-351, shape torch.Size([1, 203, 200, 200]), rank 0 
+2024-09-10 12:41:09.253998: predicting saros-357 
+2024-09-10 12:41:09.453462: saros-357, shape torch.Size([1, 177, 214, 214]), rank 0 
+2024-09-10 12:41:11.360849: predicting saros-360 
+2024-09-10 12:41:11.601134: saros-360, shape torch.Size([1, 187, 215, 215]), rank 0 
+2024-09-10 12:41:13.505020: predicting saros-365 
+2024-09-10 12:41:13.802171: saros-365, shape torch.Size([1, 240, 227, 227]), rank 0 
+2024-09-10 12:41:17.504948: predicting saros-368 
+2024-09-10 12:41:18.158620: saros-368, shape torch.Size([1, 273, 333, 333]), rank 0 
+2024-09-10 12:41:26.483354: predicting saros-370 
+2024-09-10 12:41:26.780535: saros-370, shape torch.Size([1, 233, 240, 240]), rank 0 
+2024-09-10 12:41:30.494467: predicting saros-375 
+2024-09-10 12:41:30.779261: saros-375, shape torch.Size([1, 160, 240, 240]), rank 0 
+2024-09-10 12:41:32.708078: predicting saros-380 
+2024-09-10 12:41:33.072980: saros-380, shape torch.Size([1, 227, 220, 220]), rank 0 
+2024-09-10 12:41:36.808225: predicting saros-394 
+2024-09-10 12:41:37.332286: saros-394, shape torch.Size([1, 247, 267, 267]), rank 0 
+2024-09-10 12:41:41.125561: predicting saros-397 
+2024-09-10 12:41:41.443775: saros-397, shape torch.Size([1, 190, 270, 270]), rank 0 
+2024-09-10 12:41:43.400352: predicting saros-404 
+2024-09-10 12:41:43.626534: saros-404, shape torch.Size([1, 200, 233, 233]), rank 0 
+2024-09-10 12:41:47.334725: predicting saros-409 
+2024-09-10 12:41:47.584544: saros-409, shape torch.Size([1, 193, 209, 209]), rank 0 
+2024-09-10 12:41:51.292331: predicting saros-416 
+2024-09-10 12:41:51.777346: saros-416, shape torch.Size([1, 190, 333, 333]), rank 0 
+2024-09-10 12:41:56.082215: predicting saros-417 
+2024-09-10 12:41:56.288080: saros-417, shape torch.Size([1, 187, 214, 214]), rank 0 
+2024-09-10 12:41:58.268253: predicting saros-429 
+2024-09-10 12:41:58.809011: saros-429, shape torch.Size([1, 243, 333, 333]), rank 0 
+2024-09-10 12:42:07.116747: predicting saros-430 
+2024-09-10 12:42:07.352995: saros-430, shape torch.Size([1, 213, 207, 207]), rank 0 
+2024-09-10 12:42:11.059656: predicting saros-440 
+2024-09-10 12:42:11.362317: saros-440, shape torch.Size([1, 210, 226, 226]), rank 0 
+2024-09-10 12:42:15.088258: predicting saros-451 
+2024-09-10 12:42:15.551666: saros-451, shape torch.Size([1, 223, 250, 250]), rank 0 
+2024-09-10 12:42:19.268684: predicting saros-462 
+2024-09-10 12:42:19.813826: saros-462, shape torch.Size([1, 267, 333, 333]), rank 0 
+2024-09-10 12:42:28.098276: predicting saros-464 
+2024-09-10 12:42:28.356064: saros-464, shape torch.Size([1, 237, 207, 207]), rank 0 
+2024-09-10 12:42:32.051141: predicting saros-473 
+2024-09-10 12:42:32.547144: saros-473, shape torch.Size([1, 247, 293, 293]), rank 0 
+2024-09-10 12:42:40.793848: predicting saros-482 
+2024-09-10 12:42:41.236373: saros-482, shape torch.Size([1, 247, 293, 293]), rank 0 
+2024-09-10 12:42:49.492620: predicting saros-483 
+2024-09-10 12:42:49.950558: saros-483, shape torch.Size([1, 217, 333, 333]), rank 0 
+2024-09-10 12:42:58.292600: predicting saros-486 
+2024-09-10 12:42:58.832105: saros-486, shape torch.Size([1, 210, 283, 283]), rank 0 
+2024-09-10 12:43:02.587624: predicting saros-487 
+2024-09-10 12:43:03.135922: saros-487, shape torch.Size([1, 267, 333, 333]), rank 0 
+2024-09-10 12:43:11.428249: predicting saros-499 
+2024-09-10 12:43:11.933360: saros-499, shape torch.Size([1, 243, 333, 333]), rank 0 
+2024-09-10 12:43:20.232151: predicting saros-500 
+2024-09-10 12:43:20.468812: saros-500, shape torch.Size([1, 207, 233, 233]), rank 0 
+2024-09-10 12:43:24.229628: predicting saros-507 
+2024-09-10 12:43:24.604272: saros-507, shape torch.Size([1, 203, 333, 333]), rank 0 
+2024-09-10 12:43:32.840491: predicting saros-513 
+2024-09-10 12:43:33.240469: saros-513, shape torch.Size([1, 297, 245, 245]), rank 0 
+2024-09-10 12:43:38.805643: predicting saros-518 
+2024-09-10 12:43:39.238706: saros-518, shape torch.Size([1, 230, 269, 269]), rank 0 
+2024-09-10 12:43:42.980361: predicting saros-521 
+2024-09-10 12:43:43.344361: saros-521, shape torch.Size([1, 217, 253, 253]), rank 0 
+2024-09-10 12:43:47.088903: predicting saros-526 
+2024-09-10 12:43:47.666392: saros-526, shape torch.Size([1, 253, 293, 293]), rank 0 
+2024-09-10 12:43:55.936860: predicting saros-528 
+2024-09-10 12:43:56.250699: saros-528, shape torch.Size([1, 213, 253, 253]), rank 0 
+2024-09-10 12:43:59.970832: predicting saros-539 
+2024-09-10 12:44:00.340275: saros-539, shape torch.Size([1, 230, 267, 267]), rank 0 
+2024-09-10 12:44:04.066918: predicting saros-567 
+2024-09-10 12:44:04.537412: saros-567, shape torch.Size([1, 220, 333, 333]), rank 0 
+2024-09-10 12:44:12.787854: predicting saros-569 
+2024-09-10 12:44:13.070490: saros-569, shape torch.Size([1, 190, 256, 256]), rank 0 
+2024-09-10 12:44:15.006258: predicting saros-576 
+2024-09-10 12:44:15.386593: saros-576, shape torch.Size([1, 247, 267, 267]), rank 0 
+2024-09-10 12:44:19.142874: predicting saros-578 
+2024-09-10 12:44:19.517707: saros-578, shape torch.Size([1, 253, 245, 245]), rank 0 
+2024-09-10 12:44:23.272259: predicting saros-585 
+2024-09-10 12:44:23.772959: saros-585, shape torch.Size([1, 163, 333, 333]), rank 0 
+2024-09-10 12:44:28.104301: predicting saros-591 
+2024-09-10 12:44:28.543718: saros-591, shape torch.Size([1, 180, 333, 333]), rank 0 
+2024-09-10 12:44:32.806332: predicting saros-594 
+2024-09-10 12:44:33.192243: saros-594, shape torch.Size([1, 233, 239, 239]), rank 0 
+2024-09-10 12:44:36.914104: predicting saros-599 
+2024-09-10 12:44:37.300796: saros-599, shape torch.Size([1, 233, 267, 267]), rank 0 
+2024-09-10 12:44:41.047199: predicting saros-602 
+2024-09-10 12:44:42.596228: saros-602, shape torch.Size([1, 653, 333, 333]), rank 0 
+2024-09-10 12:45:07.299498: predicting saros-613 
+2024-09-10 12:45:08.838869: saros-613, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 12:45:37.868965: predicting saros-617 
+2024-09-10 12:45:39.271357: saros-617, shape torch.Size([1, 647, 333, 333]), rank 0 
+2024-09-10 12:46:03.959875: predicting saros-618 
+2024-09-10 12:46:05.244302: saros-618, shape torch.Size([1, 593, 333, 333]), rank 0 
+2024-09-10 12:46:29.915151: predicting saros-621 
+2024-09-10 12:46:31.768408: saros-621, shape torch.Size([1, 547, 467, 467]), rank 0 
+2024-09-10 12:47:08.626675: predicting saros-629 
+2024-09-10 12:47:10.691188: saros-629, shape torch.Size([1, 660, 400, 400]), rank 0 
+2024-09-10 12:47:55.830988: predicting saros-633 
+2024-09-10 12:47:57.143032: saros-633, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-10 12:48:18.053372: predicting saros-638 
+2024-09-10 12:48:19.203965: saros-638, shape torch.Size([1, 547, 333, 333]), rank 0 
+2024-09-10 12:48:39.758882: predicting saros-640 
+2024-09-10 12:48:41.262595: saros-640, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 12:49:10.319536: predicting saros-644 
+2024-09-10 12:49:11.795057: saros-644, shape torch.Size([1, 647, 333, 333]), rank 0 
+2024-09-10 12:49:36.692882: predicting saros-653 
+2024-09-10 12:49:37.774747: saros-653, shape torch.Size([1, 497, 333, 333]), rank 0 
+2024-09-10 12:49:58.300188: predicting saros-655 
+2024-09-10 12:49:59.664117: saros-655, shape torch.Size([1, 677, 333, 333]), rank 0 
+2024-09-10 12:50:28.535355: predicting saros-659 
+2024-09-10 12:50:29.816750: saros-659, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-10 12:50:50.531069: predicting saros-665 
+2024-09-10 12:50:53.103903: saros-665, shape torch.Size([1, 1117, 333, 333]), rank 0 
+2024-09-10 12:51:38.574339: predicting saros-678 
+2024-09-10 12:51:40.110138: saros-678, shape torch.Size([1, 583, 333, 333]), rank 0 
+2024-09-10 12:52:04.938130: predicting saros-687 
+2024-09-10 12:52:06.272820: saros-687, shape torch.Size([1, 580, 333, 333]), rank 0 
+2024-09-10 12:52:30.980633: predicting saros-692 
+2024-09-10 12:52:32.220215: saros-692, shape torch.Size([1, 430, 333, 333]), rank 0 
+2024-09-10 12:52:48.706521: predicting saros-697 
+2024-09-10 12:52:50.115915: saros-697, shape torch.Size([1, 650, 333, 333]), rank 0 
+2024-09-10 12:53:14.972332: predicting saros-705 
+2024-09-10 12:53:16.692097: saros-705, shape torch.Size([1, 653, 333, 333]), rank 0 
+2024-09-10 12:53:41.924753: predicting saros-714 
+2024-09-10 12:53:44.954979: saros-714, shape torch.Size([1, 583, 333, 333]), rank 0 
+2024-09-10 12:54:09.688677: predicting saros-720 
+2024-09-10 12:54:11.093140: saros-720, shape torch.Size([1, 627, 333, 333]), rank 0 
+2024-09-10 12:54:35.811893: predicting saros-721 
+2024-09-10 12:54:37.197316: saros-721, shape torch.Size([1, 653, 333, 333]), rank 0 
+2024-09-10 12:55:01.962361: predicting saros-723 
+2024-09-10 12:55:03.133108: saros-723, shape torch.Size([1, 547, 333, 333]), rank 0 
+2024-09-10 12:55:23.763428: predicting saros-730 
+2024-09-10 12:55:25.013896: saros-730, shape torch.Size([1, 593, 333, 333]), rank 0 
+2024-09-10 12:55:49.690985: predicting saros-742 
+2024-09-10 12:55:51.041313: saros-742, shape torch.Size([1, 527, 333, 333]), rank 0 
+2024-09-10 12:56:11.763128: predicting saros-752 
+2024-09-10 12:56:13.348484: saros-752, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 12:56:42.481115: predicting saros-758 
+2024-09-10 12:56:43.839500: saros-758, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 12:57:13.007977: predicting saros-768 
+2024-09-10 12:57:14.377662: saros-768, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 12:57:43.688136: predicting saros-779 
+2024-09-10 12:57:45.830691: saros-779, shape torch.Size([1, 1180, 333, 333]), rank 0 
+2024-09-10 12:58:35.438074: predicting saros-787 
+2024-09-10 12:58:36.819927: saros-787, shape torch.Size([1, 547, 333, 333]), rank 0 
+2024-09-10 12:58:57.562515: predicting saros-802 
+2024-09-10 12:58:59.036436: saros-802, shape torch.Size([1, 660, 333, 333]), rank 0 
+2024-09-10 12:59:23.861387: predicting saros-803 
+2024-09-10 12:59:25.127130: saros-803, shape torch.Size([1, 573, 333, 333]), rank 0 
+2024-09-10 12:59:45.764300: predicting saros-808 
+2024-09-10 12:59:47.282605: saros-808, shape torch.Size([1, 727, 333, 333]), rank 0 
+2024-09-10 13:00:16.390196: predicting saros-815 
+2024-09-10 13:00:17.750358: saros-815, shape torch.Size([1, 583, 333, 333]), rank 0 
+2024-09-10 13:00:42.432550: predicting saros-823 
+2024-09-10 13:00:43.555562: saros-823, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-10 13:01:00.069578: predicting saros-828 
+2024-09-10 13:01:01.540369: saros-828, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 13:01:30.361769: predicting saros-836 
+2024-09-10 13:01:31.917630: saros-836, shape torch.Size([1, 583, 333, 333]), rank 0 
+2024-09-10 13:01:56.681763: predicting saros-841 
+2024-09-10 13:01:58.116331: saros-841, shape torch.Size([1, 650, 333, 333]), rank 0 
+2024-09-10 13:02:22.897768: predicting saros-842 
+2024-09-10 13:02:24.342128: saros-842, shape torch.Size([1, 653, 333, 333]), rank 0 
+2024-09-10 13:02:49.153936: predicting saros-844 
+2024-09-10 13:02:50.233400: saros-844, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-10 13:03:06.784205: predicting saros-845 
+2024-09-10 13:03:08.212893: saros-845, shape torch.Size([1, 647, 333, 333]), rank 0 
+2024-09-10 13:03:33.003594: predicting saros-850 
+2024-09-10 13:03:34.546664: saros-850, shape torch.Size([1, 727, 333, 333]), rank 0 
+2024-09-10 13:04:03.635909: predicting saros-861 
+2024-09-10 13:04:05.150535: saros-861, shape torch.Size([1, 647, 333, 333]), rank 0 
+2024-09-10 13:04:29.957352: predicting saros-862 
+2024-09-10 13:04:31.435339: saros-862, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 13:05:00.309417: predicting saros-863 
+2024-09-10 13:05:01.861233: saros-863, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 13:05:30.881630: predicting saros-878 
+2024-09-10 13:05:32.408070: saros-878, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 13:06:01.343868: predicting saros-886 
+2024-09-10 13:06:02.833857: saros-886, shape torch.Size([1, 673, 333, 333]), rank 0 
+2024-09-10 13:06:31.906523: predicting saros-893 
+2024-09-10 13:06:32.913841: saros-893, shape torch.Size([1, 450, 333, 333]), rank 0 
+2024-09-10 13:06:49.387379: predicting saros-894 
+2024-09-10 13:06:50.663203: saros-894, shape torch.Size([1, 653, 333, 333]), rank 0 
+2024-09-10 13:07:15.440437: predicting saros-897 
+2024-09-10 13:07:17.616784: saros-897, shape torch.Size([1, 650, 440, 440]), rank 0 
+2024-09-10 13:08:01.763572: predicting tseg-s0028 
+2024-09-10 13:08:02.265465: tseg-s0028, shape torch.Size([1, 394, 299, 299]), rank 0 
+2024-09-10 13:08:18.818898: predicting tseg-s0045 
+2024-09-10 13:08:19.102122: tseg-s0045, shape torch.Size([1, 306, 231, 231]), rank 0 
+2024-09-10 13:08:24.741470: predicting tseg-s0054 
+2024-09-10 13:08:25.096044: tseg-s0054, shape torch.Size([1, 273, 261, 261]), rank 0 
+2024-09-10 13:08:28.886979: predicting tseg-s0063 
+2024-09-10 13:08:28.992097: tseg-s0063, shape torch.Size([1, 209, 156, 209]), rank 0 
+2024-09-10 13:08:30.897987: predicting tseg-s0065 
+2024-09-10 13:08:31.306428: tseg-s0065, shape torch.Size([1, 409, 242, 242]), rank 0 
+2024-09-10 13:08:38.678081: predicting tseg-s0074 
+2024-09-10 13:08:38.822227: tseg-s0074, shape torch.Size([1, 180, 192, 251]), rank 0 
+2024-09-10 13:08:39.820775: predicting tseg-s0077 
+2024-09-10 13:08:40.212812: tseg-s0077, shape torch.Size([1, 272, 255, 255]), rank 0 
+2024-09-10 13:08:43.982980: predicting tseg-s0086 
+2024-09-10 13:08:44.672961: tseg-s0086, shape torch.Size([1, 430, 430, 273]), rank 0 
+2024-09-10 13:08:59.365632: predicting tseg-s0102 
+2024-09-10 13:08:59.641777: tseg-s0102, shape torch.Size([1, 369, 152, 369]), rank 0 
+2024-09-10 13:09:03.939281: predicting tseg-s0104 
+2024-09-10 13:09:04.240887: tseg-s0104, shape torch.Size([1, 202, 295, 295]), rank 0 
+2024-09-10 13:09:12.513465: predicting tseg-s0108 
+2024-09-10 13:09:13.004520: tseg-s0108, shape torch.Size([1, 293, 288, 288]), rank 0 
+2024-09-10 13:09:18.674138: predicting tseg-s0120 
+2024-09-10 13:09:19.090747: tseg-s0120, shape torch.Size([1, 355, 254, 261]), rank 0 
+2024-09-10 13:09:24.718786: predicting tseg-s0128 
+2024-09-10 13:09:24.784056: tseg-s0128, shape torch.Size([1, 52, 261, 261]), rank 0 
+2024-09-10 13:09:26.612399: predicting tseg-s0138 
+2024-09-10 13:09:26.820332: tseg-s0138, shape torch.Size([1, 215, 221, 221]), rank 0 
+2024-09-10 13:09:30.506569: predicting tseg-s0140 
+2024-09-10 13:09:30.584835: tseg-s0140, shape torch.Size([1, 125, 196, 196]), rank 0 
+2024-09-10 13:09:32.456442: predicting tseg-s0141 
+2024-09-10 13:09:32.889339: tseg-s0141, shape torch.Size([1, 406, 168, 406]), rank 0 
+2024-09-10 13:09:40.342833: predicting tseg-s0143 
+2024-09-10 13:09:40.548068: tseg-s0143, shape torch.Size([1, 267, 166, 279]), rank 0 
+2024-09-10 13:09:42.502724: predicting tseg-s0150 
+2024-09-10 13:09:42.895074: tseg-s0150, shape torch.Size([1, 296, 247, 247]), rank 0 
+2024-09-10 13:09:48.460680: predicting tseg-s0160 
+2024-09-10 13:09:48.609437: tseg-s0160, shape torch.Size([1, 95, 321, 321]), rank 0 
+2024-09-10 13:09:52.855851: predicting tseg-s0171 
+2024-09-10 13:09:53.208772: tseg-s0171, shape torch.Size([1, 409, 222, 222]), rank 0 
+2024-09-10 13:10:00.555918: predicting tseg-s0174 
+2024-09-10 13:10:00.672637: tseg-s0174, shape torch.Size([1, 253, 133, 194]), rank 0 
+2024-09-10 13:10:02.556416: predicting tseg-s0211 
+2024-09-10 13:10:02.659815: tseg-s0211, shape torch.Size([1, 59, 261, 261]), rank 0 
+2024-09-10 13:10:04.551345: predicting tseg-s0212 
+2024-09-10 13:10:04.840075: tseg-s0212, shape torch.Size([1, 327, 184, 262]), rank 0 
+2024-09-10 13:10:07.710683: predicting tseg-s0215 
+2024-09-10 13:10:07.788781: tseg-s0215, shape torch.Size([1, 57, 249, 249]), rank 0 
+2024-09-10 13:10:09.681077: predicting tseg-s0216 
+2024-09-10 13:10:09.917161: tseg-s0216, shape torch.Size([1, 179, 288, 288]), rank 0 
+2024-09-10 13:10:11.892837: predicting tseg-s0220 
+2024-09-10 13:10:12.131537: tseg-s0220, shape torch.Size([1, 181, 253, 253]), rank 0 
+2024-09-10 13:10:14.097182: predicting tseg-s0228 
+2024-09-10 13:10:14.398773: tseg-s0228, shape torch.Size([1, 267, 267, 254]), rank 0 
+2024-09-10 13:10:18.152603: predicting tseg-s0231 
+2024-09-10 13:10:18.332161: tseg-s0231, shape torch.Size([1, 237, 158, 248]), rank 0 
+2024-09-10 13:10:20.319211: predicting tseg-s0233 
+2024-09-10 13:10:20.347425: tseg-s0233, shape torch.Size([1, 153, 91, 91]), rank 0 
+2024-09-10 13:10:20.827332: predicting tseg-s0243 
+2024-09-10 13:10:21.141345: tseg-s0243, shape torch.Size([1, 285, 253, 253]), rank 0 
+2024-09-10 13:10:24.875197: predicting tseg-s0250 
+2024-09-10 13:10:25.421129: tseg-s0250, shape torch.Size([1, 301, 307, 307]), rank 0 
+2024-09-10 13:10:37.796527: predicting tseg-s0256 
+2024-09-10 13:10:37.935781: tseg-s0256, shape torch.Size([1, 151, 231, 231]), rank 0 
+2024-09-10 13:10:39.832496: predicting tseg-s0266 
+2024-09-10 13:10:39.998380: tseg-s0266, shape torch.Size([1, 97, 302, 302]), rank 0 
+2024-09-10 13:10:44.155386: predicting tseg-s0277 
+2024-09-10 13:10:44.291839: tseg-s0277, shape torch.Size([1, 223, 162, 223]), rank 0 
+2024-09-10 13:10:46.208797: predicting tseg-s0283 
+2024-09-10 13:10:46.303630: tseg-s0283, shape torch.Size([1, 96, 232, 232]), rank 0 
+2024-09-10 13:10:48.200531: predicting tseg-s0290 
+2024-09-10 13:10:48.362191: tseg-s0290, shape torch.Size([1, 115, 285, 285]), rank 0 
+2024-09-10 13:10:50.275250: predicting tseg-s0294 
+2024-09-10 13:10:50.391282: tseg-s0294, shape torch.Size([1, 71, 301, 301]), rank 0 
+2024-09-10 13:10:54.548351: predicting tseg-s0296 
+2024-09-10 13:10:54.559698: tseg-s0296, shape torch.Size([1, 60, 80, 80]), rank 0 
+2024-09-10 13:10:55.035226: predicting tseg-s0324 
+2024-09-10 13:10:55.235258: tseg-s0324, shape torch.Size([1, 261, 200, 200]), rank 0 
+2024-09-10 13:10:58.904962: predicting tseg-s0333 
+2024-09-10 13:10:59.075521: tseg-s0333, shape torch.Size([1, 217, 219, 219]), rank 0 
+2024-09-10 13:11:02.780535: predicting tseg-s0334 
+2024-09-10 13:11:03.393595: tseg-s0334, shape torch.Size([1, 469, 291, 291]), rank 0 
+2024-09-10 13:11:19.818583: predicting tseg-s0341 
+2024-09-10 13:11:20.084182: tseg-s0341, shape torch.Size([1, 275, 247, 247]), rank 0 
+2024-09-10 13:11:23.902013: predicting tseg-s0344 
+2024-09-10 13:11:24.418565: tseg-s0344, shape torch.Size([1, 384, 283, 283]), rank 0 
+2024-09-10 13:11:30.074203: predicting tseg-s0356 
+2024-09-10 13:11:30.100876: tseg-s0356, shape torch.Size([1, 89, 107, 107]), rank 0 
+2024-09-10 13:11:30.590909: predicting tseg-s0357 
+2024-09-10 13:11:30.787707: tseg-s0357, shape torch.Size([1, 215, 223, 223]), rank 0 
+2024-09-10 13:11:34.467418: predicting tseg-s0360 
+2024-09-10 13:11:34.524705: tseg-s0360, shape torch.Size([1, 53, 253, 253]), rank 0 
+2024-09-10 13:11:36.385646: predicting tseg-s0363 
+2024-09-10 13:11:36.588720: tseg-s0363, shape torch.Size([1, 208, 228, 228]), rank 0 
+2024-09-10 13:11:40.284648: predicting tseg-s0372 
+2024-09-10 13:11:40.634818: tseg-s0372, shape torch.Size([1, 295, 256, 256]), rank 0 
+2024-09-10 13:11:46.203086: predicting tseg-s0392 
+2024-09-10 13:11:46.535606: tseg-s0392, shape torch.Size([1, 270, 267, 267]), rank 0 
+2024-09-10 13:11:50.334939: predicting tseg-s0403 
+2024-09-10 13:11:50.835933: tseg-s0403, shape torch.Size([1, 391, 267, 267]), rank 0 
+2024-09-10 13:11:58.278367: predicting tseg-s0405 
+2024-09-10 13:11:58.408896: tseg-s0405, shape torch.Size([1, 182, 196, 196]), rank 0 
+2024-09-10 13:12:00.294005: predicting tseg-s0407 
+2024-09-10 13:12:00.526701: tseg-s0407, shape torch.Size([1, 228, 215, 215]), rank 0 
+2024-09-10 13:12:04.229506: predicting tseg-s0414 
+2024-09-10 13:12:04.369434: tseg-s0414, shape torch.Size([1, 171, 199, 199]), rank 0 
+2024-09-10 13:12:06.269614: predicting tseg-s0425 
+2024-09-10 13:12:06.541636: tseg-s0425, shape torch.Size([1, 196, 260, 260]), rank 0 
+2024-09-10 13:12:10.260618: predicting tseg-s0428 
+2024-09-10 13:12:10.730385: tseg-s0428, shape torch.Size([1, 317, 307, 307]), rank 0 
+2024-09-10 13:12:23.079827: predicting tseg-s0429 
+2024-09-10 13:12:23.533924: tseg-s0429, shape torch.Size([1, 433, 254, 254]), rank 0 
+2024-09-10 13:12:31.008829: predicting tseg-s0439 
+2024-09-10 13:12:31.418005: tseg-s0439, shape torch.Size([1, 334, 221, 307]), rank 0 
+2024-09-10 13:12:39.752917: predicting tseg-s0442 
+2024-09-10 13:12:39.981934: tseg-s0442, shape torch.Size([1, 178, 264, 264]), rank 0 
+2024-09-10 13:12:41.935952: predicting tseg-s0461 
+2024-09-10 13:12:42.498417: tseg-s0461, shape torch.Size([1, 447, 269, 269]), rank 0 
+2024-09-10 13:12:49.934301: predicting tseg-s0466 
+2024-09-10 13:12:50.005198: tseg-s0466, shape torch.Size([1, 143, 173, 173]), rank 0 
+2024-09-10 13:12:50.528234: predicting tseg-s0471 
+2024-09-10 13:12:50.672365: tseg-s0471, shape torch.Size([1, 142, 227, 227]), rank 0 
+2024-09-10 13:12:52.612878: predicting tseg-s0473 
+2024-09-10 13:12:52.898112: tseg-s0473, shape torch.Size([1, 289, 242, 242]), rank 0 
+2024-09-10 13:12:58.413717: predicting tseg-s0481 
+2024-09-10 13:12:58.587977: tseg-s0481, shape torch.Size([1, 185, 207, 207]), rank 0 
+2024-09-10 13:13:00.528398: predicting tseg-s0495 
+2024-09-10 13:13:00.894602: tseg-s0495, shape torch.Size([1, 332, 247, 247]), rank 0 
+2024-09-10 13:13:06.462233: predicting tseg-s0500 
+2024-09-10 13:13:07.019562: tseg-s0500, shape torch.Size([1, 354, 307, 307]), rank 0 
+2024-09-10 13:13:19.404243: predicting tseg-s0502 
+2024-09-10 13:13:19.757299: tseg-s0502, shape torch.Size([1, 402, 215, 215]), rank 0 
+2024-09-10 13:13:27.163871: predicting tseg-s0515 
+2024-09-10 13:13:27.304413: tseg-s0515, shape torch.Size([1, 176, 205, 205]), rank 0 
+2024-09-10 13:13:29.249837: predicting tseg-s0518 
+2024-09-10 13:13:29.353481: tseg-s0518, shape torch.Size([1, 236, 145, 145]), rank 0 
+2024-09-10 13:13:30.334086: predicting tseg-s0519 
+2024-09-10 13:13:30.906770: tseg-s0519, shape torch.Size([1, 433, 283, 283]), rank 0 
+2024-09-10 13:13:38.356550: predicting tseg-s0522 
+2024-09-10 13:13:38.558709: tseg-s0522, shape torch.Size([1, 199, 249, 249]), rank 0 
+2024-09-10 13:13:42.347908: predicting tseg-s0525 
+2024-09-10 13:13:42.418925: tseg-s0525, shape torch.Size([1, 56, 297, 297]), rank 0 
+2024-09-10 13:13:46.559601: predicting tseg-s0529 
+2024-09-10 13:13:46.903960: tseg-s0529, shape torch.Size([1, 304, 247, 247]), rank 0 
+2024-09-10 13:13:52.444678: predicting tseg-s0545 
+2024-09-10 13:13:52.741713: tseg-s0545, shape torch.Size([1, 240, 257, 257]), rank 0 
+2024-09-10 13:13:56.514254: predicting tseg-s0553 
+2024-09-10 13:13:56.887577: tseg-s0553, shape torch.Size([1, 414, 233, 233]), rank 0 
+2024-09-10 13:14:04.267659: predicting tseg-s0556 
+2024-09-10 13:14:05.054357: tseg-s0556, shape torch.Size([1, 484, 333, 333]), rank 0 
+2024-09-10 13:14:25.638755: predicting tseg-s0565 
+2024-09-10 13:14:25.841482: tseg-s0565, shape torch.Size([1, 204, 245, 245]), rank 0 
+2024-09-10 13:14:29.580484: predicting tseg-s0566 
+2024-09-10 13:14:29.901909: tseg-s0566, shape torch.Size([1, 319, 161, 319]), rank 0 
+2024-09-10 13:14:34.166824: predicting tseg-s0567 
+2024-09-10 13:14:34.215038: tseg-s0567, shape torch.Size([1, 112, 161, 161]), rank 0 
+2024-09-10 13:14:34.722493: predicting tseg-s0572 
+2024-09-10 13:14:34.805659: tseg-s0572, shape torch.Size([1, 133, 185, 185]), rank 0 
+2024-09-10 13:14:35.323714: predicting tseg-s0574 
+2024-09-10 13:14:35.840222: tseg-s0574, shape torch.Size([1, 441, 251, 251]), rank 0 
+2024-09-10 13:14:43.217476: predicting tseg-s0575 
+2024-09-10 13:14:43.255105: tseg-s0575, shape torch.Size([1, 103, 143, 143]), rank 0 
+2024-09-10 13:14:43.744535: predicting tseg-s0578 
+2024-09-10 13:14:44.130836: tseg-s0578, shape torch.Size([1, 395, 223, 223]), rank 0 
+2024-09-10 13:14:51.472107: predicting tseg-s0579 
+2024-09-10 13:14:51.545609: tseg-s0579, shape torch.Size([1, 213, 128, 128]), rank 0 
+2024-09-10 13:14:52.505121: predicting tseg-s0580 
+2024-09-10 13:14:52.810101: tseg-s0580, shape torch.Size([1, 288, 181, 288]), rank 0 
+2024-09-10 13:14:54.768904: predicting tseg-s0581 
+2024-09-10 13:14:54.988677: tseg-s0581, shape torch.Size([1, 198, 233, 233]), rank 0 
+2024-09-10 13:14:58.714020: predicting tseg-s0587 
+2024-09-10 13:14:59.276187: tseg-s0587, shape torch.Size([1, 410, 300, 300]), rank 0 
+2024-09-10 13:15:15.674031: predicting tseg-s0589 
+2024-09-10 13:15:16.212518: tseg-s0589, shape torch.Size([1, 423, 265, 265]), rank 0 
+2024-09-10 13:15:23.692127: predicting tseg-s0590 
+2024-09-10 13:15:23.874649: tseg-s0590, shape torch.Size([1, 260, 192, 260]), rank 0 
+2024-09-10 13:15:25.804942: predicting tseg-s0594 
+2024-09-10 13:15:25.943519: tseg-s0594, shape torch.Size([1, 121, 216, 216]), rank 0 
+2024-09-10 13:15:27.834215: predicting tseg-s0601 
+2024-09-10 13:15:27.972707: tseg-s0601, shape torch.Size([1, 150, 218, 218]), rank 0 
+2024-09-10 13:15:29.861240: predicting tseg-s0604 
+2024-09-10 13:15:30.039398: tseg-s0604, shape torch.Size([1, 208, 199, 199]), rank 0 
+2024-09-10 13:15:33.717805: predicting tseg-s0610 
+2024-09-10 13:15:33.929014: tseg-s0610, shape torch.Size([1, 206, 240, 240]), rank 0 
+2024-09-10 13:15:37.627915: predicting tseg-s0612 
+2024-09-10 13:15:38.176778: tseg-s0612, shape torch.Size([1, 411, 272, 272]), rank 0 
+2024-09-10 13:15:45.607896: predicting tseg-s0613 
+2024-09-10 13:15:46.186995: tseg-s0613, shape torch.Size([1, 417, 272, 272]), rank 0 
+2024-09-10 13:15:53.676399: predicting tseg-s0620 
+2024-09-10 13:15:54.060115: tseg-s0620, shape torch.Size([1, 300, 244, 244]), rank 0 
+2024-09-10 13:15:59.664978: predicting tseg-s0627 
+2024-09-10 13:15:59.856832: tseg-s0627, shape torch.Size([1, 157, 264, 264]), rank 0 
+2024-09-10 13:16:01.815063: predicting tseg-s0643 
+2024-09-10 13:16:01.987832: tseg-s0643, shape torch.Size([1, 263, 178, 178]), rank 0 
+2024-09-10 13:16:03.003422: predicting tseg-s0644 
+2024-09-10 13:16:03.480129: tseg-s0644, shape torch.Size([1, 389, 272, 272]), rank 0 
+2024-09-10 13:16:10.877111: predicting tseg-s0654 
+2024-09-10 13:16:11.079732: tseg-s0654, shape torch.Size([1, 205, 222, 222]), rank 0 
+2024-09-10 13:16:14.842746: predicting tseg-s0655 
+2024-09-10 13:16:14.862959: tseg-s0655, shape torch.Size([1, 92, 95, 95]), rank 0 
+2024-09-10 13:16:15.340579: predicting tseg-s0661 
+2024-09-10 13:16:15.774084: tseg-s0661, shape torch.Size([1, 390, 252, 252]), rank 0 
+2024-09-10 13:16:23.124087: predicting tseg-s0675 
+2024-09-10 13:16:23.482658: tseg-s0675, shape torch.Size([1, 209, 309, 309]), rank 0 
+2024-09-10 13:16:31.782362: predicting tseg-s0686 
+2024-09-10 13:16:32.290437: tseg-s0686, shape torch.Size([1, 407, 265, 265]), rank 0 
+2024-09-10 13:16:39.735724: predicting tseg-s0714 
+2024-09-10 13:16:39.960831: tseg-s0714, shape torch.Size([1, 183, 262, 262]), rank 0 
+2024-09-10 13:16:41.911601: predicting tseg-s0716 
+2024-09-10 13:16:42.153174: tseg-s0716, shape torch.Size([1, 228, 238, 238]), rank 0 
+2024-09-10 13:16:45.879828: predicting tseg-s0721 
+2024-09-10 13:16:46.135558: tseg-s0721, shape torch.Size([1, 265, 227, 227]), rank 0 
+2024-09-10 13:16:49.865994: predicting tseg-s0738 
+2024-09-10 13:16:49.915123: tseg-s0738, shape torch.Size([1, 141, 119, 119]), rank 0 
+2024-09-10 13:16:50.401254: predicting tseg-s0764 
+2024-09-10 13:16:50.998100: tseg-s0764, shape torch.Size([1, 405, 291, 291]), rank 0 
+2024-09-10 13:17:07.361961: predicting tseg-s0771 
+2024-09-10 13:17:07.764560: tseg-s0771, shape torch.Size([1, 247, 310, 310]), rank 0 
+2024-09-10 13:17:16.103611: predicting tseg-s0778 
+2024-09-10 13:17:16.634349: tseg-s0778, shape torch.Size([1, 431, 262, 262]), rank 0 
+2024-09-10 13:17:24.099394: predicting tseg-s0785 
+2024-09-10 13:17:24.238564: tseg-s0785, shape torch.Size([1, 211, 179, 209]), rank 0 
+2024-09-10 13:17:26.134026: predicting tseg-s0786 
+2024-09-10 13:17:26.236464: tseg-s0786, shape torch.Size([1, 155, 174, 174]), rank 0 
+2024-09-10 13:17:26.766490: predicting tseg-s0788 
+2024-09-10 13:17:27.033212: tseg-s0788, shape torch.Size([1, 267, 225, 225]), rank 0 
+2024-09-10 13:17:30.747708: predicting tseg-s0789 
+2024-09-10 13:17:31.077266: tseg-s0789, shape torch.Size([1, 242, 279, 279]), rank 0 
+2024-09-10 13:17:34.855817: predicting tseg-s0810 
+2024-09-10 13:17:35.278612: tseg-s0810, shape torch.Size([1, 289, 289, 289]), rank 0 
+2024-09-10 13:17:47.629306: predicting tseg-s0812 
+2024-09-10 13:17:48.688504: tseg-s0812, shape torch.Size([1, 549, 327, 327]), rank 0 
+2024-09-10 13:18:09.337291: predicting tseg-s0815 
+2024-09-10 13:18:09.587710: tseg-s0815, shape torch.Size([1, 267, 253, 253]), rank 0 
+2024-09-10 13:18:13.349431: predicting tseg-s0824 
+2024-09-10 13:18:13.561021: tseg-s0824, shape torch.Size([1, 200, 183, 265]), rank 0 
+2024-09-10 13:18:15.511178: predicting tseg-s0831 
+2024-09-10 13:18:16.026786: tseg-s0831, shape torch.Size([1, 524, 253, 253]), rank 0 
+2024-09-10 13:18:25.246438: predicting tseg-s0832 
+2024-09-10 13:18:25.365604: tseg-s0832, shape torch.Size([1, 104, 257, 257]), rank 0 
+2024-09-10 13:18:27.249902: predicting tseg-s0863 
+2024-09-10 13:18:27.768467: tseg-s0863, shape torch.Size([1, 435, 263, 263]), rank 0 
+2024-09-10 13:18:35.191302: predicting tseg-s0869 
+2024-09-10 13:18:35.733908: tseg-s0869, shape torch.Size([1, 459, 261, 261]), rank 0 
+2024-09-10 13:18:43.231477: predicting tseg-s0876 
+2024-09-10 13:18:43.260825: tseg-s0876, shape torch.Size([1, 95, 119, 119]), rank 0 
+2024-09-10 13:18:43.759232: predicting tseg-s0880 
+2024-09-10 13:18:44.353836: tseg-s0880, shape torch.Size([1, 463, 291, 291]), rank 0 
+2024-09-10 13:19:00.756094: predicting tseg-s0884 
+2024-09-10 13:19:01.384492: tseg-s0884, shape torch.Size([1, 455, 272, 272]), rank 0 
+2024-09-10 13:19:08.930197: predicting tseg-s0889 
+2024-09-10 13:19:09.157362: tseg-s0889, shape torch.Size([1, 203, 251, 251]), rank 0 
+2024-09-10 13:19:12.941825: predicting tseg-s0891 
+2024-09-10 13:19:12.981162: tseg-s0891, shape torch.Size([1, 131, 117, 117]), rank 0 
+2024-09-10 13:19:13.462410: predicting tseg-s0908 
+2024-09-10 13:19:13.698482: tseg-s0908, shape torch.Size([1, 180, 261, 261]), rank 0 
+2024-09-10 13:19:15.644192: predicting tseg-s0913 
+2024-09-10 13:19:16.289337: tseg-s0913, shape torch.Size([1, 465, 289, 289]), rank 0 
+2024-09-10 13:19:32.717375: predicting tseg-s0943 
+2024-09-10 13:19:33.011103: tseg-s0943, shape torch.Size([1, 252, 289, 289]), rank 0 
+2024-09-10 13:19:41.379472: predicting tseg-s0945 
+2024-09-10 13:19:41.933556: tseg-s0945, shape torch.Size([1, 399, 285, 285]), rank 0 
+2024-09-10 13:19:49.384325: predicting tseg-s0946 
+2024-09-10 13:19:49.583113: tseg-s0946, shape torch.Size([1, 212, 236, 236]), rank 0 
+2024-09-10 13:19:53.348772: predicting tseg-s0957 
+2024-09-10 13:19:53.818739: tseg-s0957, shape torch.Size([1, 437, 248, 248]), rank 0 
+2024-09-10 13:20:01.233418: predicting tseg-s0970 
+2024-09-10 13:20:02.050682: tseg-s0970, shape torch.Size([1, 511, 333, 333]), rank 0 
+2024-09-10 13:20:22.683846: predicting tseg-s0979 
+2024-09-10 13:20:23.088574: tseg-s0979, shape torch.Size([1, 349, 193, 432]), rank 0 
+2024-09-10 13:20:34.218049: predicting tseg-s0981 
+2024-09-10 13:20:34.373030: tseg-s0981, shape torch.Size([1, 174, 221, 221]), rank 0 
+2024-09-10 13:20:36.259832: predicting tseg-s0997 
+2024-09-10 13:20:36.296472: tseg-s0997, shape torch.Size([1, 96, 137, 137]), rank 0 
+2024-09-10 13:20:36.780007: predicting tseg-s1010 
+2024-09-10 13:20:36.909846: tseg-s1010, shape torch.Size([1, 169, 208, 208]), rank 0 
+2024-09-10 13:20:38.816072: predicting tseg-s1014 
+2024-09-10 13:20:38.848826: tseg-s1014, shape torch.Size([1, 116, 133, 111]), rank 0 
+2024-09-10 13:20:39.340490: predicting tseg-s1021 
+2024-09-10 13:20:39.455910: tseg-s1021, shape torch.Size([1, 136, 136, 136]), rank 0 
+2024-09-10 13:20:39.949530: predicting tseg-s1022 
+2024-09-10 13:20:40.400983: tseg-s1022, shape torch.Size([1, 440, 236, 236]), rank 0 
+2024-09-10 13:20:47.764235: predicting tseg-s1028 
+2024-09-10 13:20:48.054185: tseg-s1028, shape torch.Size([1, 282, 232, 232]), rank 0 
+2024-09-10 13:20:51.855031: predicting tseg-s1037 
+2024-09-10 13:20:52.159924: tseg-s1037, shape torch.Size([1, 285, 216, 250]), rank 0 
+2024-09-10 13:20:55.919836: predicting tseg-s1043 
+2024-09-10 13:20:55.990260: tseg-s1043, shape torch.Size([1, 157, 164, 164]), rank 0 
+2024-09-10 13:20:56.504504: predicting tseg-s1055 
+2024-09-10 13:20:56.541001: tseg-s1055, shape torch.Size([1, 70, 171, 171]), rank 0 
+2024-09-10 13:20:57.040486: predicting tseg-s1068 
+2024-09-10 13:20:57.134644: tseg-s1068, shape torch.Size([1, 80, 223, 223]), rank 0 
+2024-09-10 13:20:59.006423: predicting tseg-s1089 
+2024-09-10 13:20:59.683665: tseg-s1089, shape torch.Size([1, 550, 265, 265]), rank 0 
+2024-09-10 13:21:08.911531: predicting tseg-s1100 
+2024-09-10 13:21:09.183090: tseg-s1100, shape torch.Size([1, 286, 231, 231]), rank 0 
+2024-09-10 13:21:12.990459: predicting tseg-s1106 
+2024-09-10 13:21:13.016589: tseg-s1106, shape torch.Size([1, 92, 110, 110]), rank 0 
+2024-09-10 13:21:13.499974: predicting tseg-s1113 
+2024-09-10 13:21:13.683656: tseg-s1113, shape torch.Size([1, 219, 206, 206]), rank 0 
+2024-09-10 13:21:17.359551: predicting tseg-s1122 
+2024-09-10 13:21:17.401614: tseg-s1122, shape torch.Size([1, 156, 114, 114]), rank 0 
+2024-09-10 13:21:17.880599: predicting tseg-s1123 
+2024-09-10 13:21:18.615965: tseg-s1123, shape torch.Size([1, 545, 291, 291]), rank 0 
+2024-09-10 13:21:39.088234: predicting tseg-s1155 
+2024-09-10 13:21:39.337158: tseg-s1155, shape torch.Size([1, 201, 271, 271]), rank 0 
+2024-09-10 13:21:43.087080: predicting tseg-s1157 
+2024-09-10 13:21:43.331696: tseg-s1157, shape torch.Size([1, 189, 253, 253]), rank 0 
+2024-09-10 13:21:45.308692: predicting tseg-s1169 
+2024-09-10 13:21:45.361249: tseg-s1169, shape torch.Size([1, 106, 151, 151]), rank 0 
+2024-09-10 13:21:45.843914: predicting tseg-s1183 
+2024-09-10 13:21:46.204071: tseg-s1183, shape torch.Size([1, 335, 216, 254]), rank 0 
+2024-09-10 13:21:51.754401: predicting tseg-s1203 
+2024-09-10 13:21:51.846997: tseg-s1203, shape torch.Size([1, 133, 183, 183]), rank 0 
+2024-09-10 13:21:52.370925: predicting tseg-s1206 
+2024-09-10 13:21:53.153483: tseg-s1206, shape torch.Size([1, 538, 279, 279]), rank 0 
+2024-09-10 13:22:02.419373: predicting tseg-s1220 
+2024-09-10 13:22:02.515095: tseg-s1220, shape torch.Size([1, 163, 198, 198]), rank 0 
+2024-09-10 13:22:04.393296: predicting tseg-s1227 
+2024-09-10 13:22:04.435395: tseg-s1227, shape torch.Size([1, 98, 165, 165]), rank 0 
+2024-09-10 13:22:04.940524: predicting tseg-s1228 
+2024-09-10 13:22:05.734110: tseg-s1228, shape torch.Size([1, 402, 321, 321]), rank 0 
+2024-09-10 13:22:22.167756: predicting tseg-s1231 
+2024-09-10 13:22:22.281830: tseg-s1231, shape torch.Size([1, 229, 177, 177]), rank 0 
+2024-09-10 13:22:23.308969: predicting tseg-s1236 
+2024-09-10 13:22:23.582358: tseg-s1236, shape torch.Size([1, 136, 325, 325]), rank 0 
+2024-09-10 13:22:27.874378: predicting tseg-s1246 
+2024-09-10 13:22:27.969404: tseg-s1246, shape torch.Size([1, 64, 297, 297]), rank 0 
+2024-09-10 13:22:32.117252: predicting tseg-s1256 
+2024-09-10 13:22:32.457185: tseg-s1256, shape torch.Size([1, 228, 292, 292]), rank 0 
+2024-09-10 13:22:40.697243: predicting tseg-s1258 
+2024-09-10 13:22:40.809024: tseg-s1258, shape torch.Size([1, 122, 221, 221]), rank 0 
+2024-09-10 13:22:42.683892: predicting tseg-s1262 
+2024-09-10 13:22:42.977808: tseg-s1262, shape torch.Size([1, 274, 221, 221]), rank 0 
+2024-09-10 13:22:46.695786: predicting tseg-s1270 
+2024-09-10 13:22:46.926581: tseg-s1270, shape torch.Size([1, 209, 233, 233]), rank 0 
+2024-09-10 13:22:50.635095: predicting tseg-s1277 
+2024-09-10 13:22:50.817372: tseg-s1277, shape torch.Size([1, 164, 251, 251]), rank 0 
+2024-09-10 13:22:52.746039: predicting tseg-s1290 
+2024-09-10 13:22:52.770205: tseg-s1290, shape torch.Size([1, 32, 176, 176]), rank 0 
+2024-09-10 13:22:53.245815: predicting tseg-s1294 
+2024-09-10 13:22:53.517237: tseg-s1294, shape torch.Size([1, 278, 238, 238]), rank 0 
+2024-09-10 13:22:57.223475: predicting tseg-s1298 
+2024-09-10 13:22:57.608674: tseg-s1298, shape torch.Size([1, 237, 297, 297]), rank 0 
+2024-09-10 13:23:05.883107: predicting tseg-s1311 
+2024-09-10 13:23:06.341203: tseg-s1311, shape torch.Size([1, 289, 288, 288]), rank 0 
+2024-09-10 13:23:11.957228: predicting tseg-s1336 
+2024-09-10 13:23:12.577431: tseg-s1336, shape torch.Size([1, 416, 285, 285]), rank 0 
+2024-09-10 13:23:20.083518: predicting tseg-s1350 
+2024-09-10 13:23:21.043629: tseg-s1350, shape torch.Size([1, 539, 333, 333]), rank 0 
+2024-09-10 13:23:41.675411: predicting tseg-s1353 
+2024-09-10 13:23:41.937196: tseg-s1353, shape torch.Size([1, 212, 249, 249]), rank 0 
+2024-09-10 13:23:45.694959: predicting tseg-s1359 
+2024-09-10 13:23:45.777134: tseg-s1359, shape torch.Size([1, 196, 140, 140]), rank 0 
+2024-09-10 13:23:46.760756: predicting tseg-s1361 
+2024-09-10 13:23:47.382580: tseg-s1361, shape torch.Size([1, 475, 283, 283]), rank 0 
+2024-09-10 13:23:54.832520: predicting tseg-s1387 
+2024-09-10 13:23:55.460428: tseg-s1387, shape torch.Size([1, 453, 281, 281]), rank 0 
+2024-09-10 13:24:03.010964: predicting tseg-s1390 
+2024-09-10 13:24:03.533559: tseg-s1390, shape torch.Size([1, 335, 285, 285]), rank 0 
+2024-09-10 13:24:09.234885: predicting tseg-s1394 
+2024-09-10 13:24:09.644821: tseg-s1394, shape torch.Size([1, 306, 261, 261]), rank 0 
+2024-09-10 13:25:42.150737: Validation complete 
+2024-09-10 13:25:42.151508: Mean Validation Dice:  0.945008093016126 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_4_16_32_31.txt

@@ -0,0 +1,803 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-04 16:32:31.301408: do_dummy_2d_data_aug: False 
+2024-09-04 16:32:31.307724: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-04 16:32:31.309105: The split file contains 5 splits. 
+2024-09-04 16:32:31.309533: Desired fold for training: 1 
+2024-09-04 16:32:31.309917: This split has 1325 training and 331 validation cases. 
+2024-09-04 16:32:48.052401: Using torch.compile... 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-04 16:32:51.086858: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-04 16:32:52.927613:  
+2024-09-04 16:32:52.932694: Epoch 0 
+2024-09-04 16:32:52.942069: Current learning rate: 0.01 
+2024-09-04 16:51:22.141526: train_loss 0.8145 
+2024-09-04 16:51:22.147962: val_loss 0.4547 
+2024-09-04 16:51:22.148629: Pseudo dice [0.552, 0.6481, 0.468, 0.0, 0.0, 0.0, 0.7819] 
+2024-09-04 16:51:22.149359: Epoch time: 1109.22 s 
+2024-09-04 16:51:22.150030: Yayy! New best EMA pseudo Dice: 0.35 
+2024-09-04 16:51:33.198290:  
+2024-09-04 16:51:33.199123: Epoch 1 
+2024-09-04 16:51:33.199790: Current learning rate: 0.00999 
+2024-09-04 16:55:47.720632: train_loss 0.3023 
+2024-09-04 16:55:47.726296: val_loss 0.0981 
+2024-09-04 16:55:47.727041: Pseudo dice [0.7345, 0.6946, 0.7738, 0.6432, 0.0, 0.0, 0.9011] 
+2024-09-04 16:55:47.727960: Epoch time: 254.53 s 
+2024-09-04 16:55:47.728801: Yayy! New best EMA pseudo Dice: 0.3685 
+2024-09-04 16:55:55.881844:  
+2024-09-04 16:55:55.883279: Epoch 2 
+2024-09-04 16:55:55.884563: Current learning rate: 0.00998 
+2024-09-04 16:59:55.501038: train_loss 0.02 
+2024-09-04 16:59:55.503590: val_loss -0.0426 
+2024-09-04 16:59:55.504542: Pseudo dice [0.7627, 0.7588, 0.8304, 0.7126, 0.0028, 0.2477, 0.9027] 
+2024-09-04 16:59:55.505362: Epoch time: 239.62 s 
+2024-09-04 16:59:55.506061: Yayy! New best EMA pseudo Dice: 0.3919 
+2024-09-04 17:00:03.753970:  
+2024-09-04 17:00:03.754830: Epoch 3 
+2024-09-04 17:00:03.755399: Current learning rate: 0.00997 
+2024-09-04 17:04:11.310206: train_loss -0.1274 
+2024-09-04 17:04:11.314772: val_loss -0.2806 
+2024-09-04 17:04:11.316435: Pseudo dice [0.8281, 0.8112, 0.8881, 0.7738, 0.1976, 0.6395, 0.9427] 
+2024-09-04 17:04:11.317780: Epoch time: 247.56 s 
+2024-09-04 17:04:11.318952: Yayy! New best EMA pseudo Dice: 0.4253 
+2024-09-04 17:04:19.048672:  
+2024-09-04 17:04:19.049675: Epoch 4 
+2024-09-04 17:04:19.050588: Current learning rate: 0.00996 
+2024-09-04 17:08:28.612441: train_loss -0.2639 
+2024-09-04 17:08:28.627337: val_loss -0.3494 
+2024-09-04 17:08:28.628248: Pseudo dice [0.8375, 0.8252, 0.8945, 0.8002, 0.5543, 0.7422, 0.9557] 
+2024-09-04 17:08:28.629013: Epoch time: 249.57 s 
+2024-09-04 17:08:28.629661: Yayy! New best EMA pseudo Dice: 0.4629 
+2024-09-04 17:08:37.269554:  
+2024-09-04 17:08:37.270661: Epoch 5 
+2024-09-04 17:08:37.271667: Current learning rate: 0.00995 
+2024-09-04 17:12:48.640265: train_loss -0.3649 
+2024-09-04 17:12:48.643614: val_loss -0.3828 
+2024-09-04 17:12:48.644808: Pseudo dice [0.8551, 0.8498, 0.8965, 0.8283, 0.6273, 0.7781, 0.9459] 
+2024-09-04 17:12:48.645692: Epoch time: 251.37 s 
+2024-09-04 17:12:48.646511: Yayy! New best EMA pseudo Dice: 0.4992 
+2024-09-04 17:12:55.624857:  
+2024-09-04 17:12:55.625726: Epoch 6 
+2024-09-04 17:12:55.626498: Current learning rate: 0.00995 
+2024-09-04 17:17:07.804024: train_loss -0.387 
+2024-09-04 17:17:07.823037: val_loss -0.4663 
+2024-09-04 17:17:07.823791: Pseudo dice [0.879, 0.8676, 0.9189, 0.8534, 0.6841, 0.7891, 0.9692] 
+2024-09-04 17:17:07.824573: Epoch time: 252.18 s 
+2024-09-04 17:17:07.825217: Yayy! New best EMA pseudo Dice: 0.5345 
+2024-09-04 17:17:16.133538:  
+2024-09-04 17:17:16.135384: Epoch 7 
+2024-09-04 17:17:16.137033: Current learning rate: 0.00994 
+2024-09-04 17:21:38.763865: train_loss -0.424 
+2024-09-04 17:21:38.779323: val_loss -0.4048 
+2024-09-04 17:21:38.780288: Pseudo dice [0.8613, 0.8703, 0.901, 0.8493, 0.6871, 0.8112, 0.9641] 
+2024-09-04 17:21:38.781196: Epoch time: 262.64 s 
+2024-09-04 17:21:38.782130: Yayy! New best EMA pseudo Dice: 0.5659 
+2024-09-04 17:21:46.929882:  
+2024-09-04 17:21:46.931554: Epoch 8 
+2024-09-04 17:21:46.932648: Current learning rate: 0.00993 
+2024-09-04 17:26:08.777971: train_loss -0.455 
+2024-09-04 17:26:08.791963: val_loss -0.462 
+2024-09-04 17:26:08.792798: Pseudo dice [0.8794, 0.8727, 0.9145, 0.858, 0.6664, 0.8372, 0.9578] 
+2024-09-04 17:26:08.793536: Epoch time: 261.85 s 
+2024-09-04 17:26:08.794872: Yayy! New best EMA pseudo Dice: 0.5949 
+2024-09-04 17:26:16.464414:  
+2024-09-04 17:26:16.465279: Epoch 9 
+2024-09-04 17:26:16.465895: Current learning rate: 0.00992 
+2024-09-04 17:30:27.501481: train_loss -0.4978 
+2024-09-04 17:30:27.504099: val_loss -0.5295 
+2024-09-04 17:30:27.504816: Pseudo dice [0.8946, 0.8987, 0.9272, 0.8686, 0.7083, 0.8831, 0.975] 
+2024-09-04 17:30:27.505904: Epoch time: 251.04 s 
+2024-09-04 17:30:27.506672: Yayy! New best EMA pseudo Dice: 0.6233 
+2024-09-04 17:30:35.890127:  
+2024-09-04 17:30:35.891496: Epoch 10 
+2024-09-04 17:30:35.892809: Current learning rate: 0.00991 
+2024-09-04 17:34:43.641668: train_loss -0.5048 
+2024-09-04 17:34:43.643223: val_loss -0.4738 
+2024-09-04 17:34:43.644495: Pseudo dice [0.8834, 0.8851, 0.9131, 0.8502, 0.7267, 0.8533, 0.9673] 
+2024-09-04 17:34:43.646155: Epoch time: 247.76 s 
+2024-09-04 17:34:43.647396: Yayy! New best EMA pseudo Dice: 0.6478 
+2024-09-04 17:34:53.053837:  
+2024-09-04 17:34:53.054836: Epoch 11 
+2024-09-04 17:34:53.055565: Current learning rate: 0.0099 
+2024-09-04 17:38:53.170877: train_loss -0.5179 
+2024-09-04 17:38:53.174992: val_loss -0.5631 
+2024-09-04 17:38:53.176498: Pseudo dice [0.9072, 0.902, 0.9387, 0.8807, 0.7307, 0.8889, 0.9748] 
+2024-09-04 17:38:53.177814: Epoch time: 240.12 s 
+2024-09-04 17:38:53.178964: Yayy! New best EMA pseudo Dice: 0.6719 
+2024-09-04 17:39:00.840950:  
+2024-09-04 17:39:00.842179: Epoch 12 
+2024-09-04 17:39:00.843600: Current learning rate: 0.00989 
+2024-09-04 17:43:15.768178: train_loss -0.5207 
+2024-09-04 17:43:15.784092: val_loss -0.5283 
+2024-09-04 17:43:15.785001: Pseudo dice [0.9075, 0.9094, 0.9402, 0.8897, 0.7255, 0.8841, 0.9426] 
+2024-09-04 17:43:15.786054: Epoch time: 254.93 s 
+2024-09-04 17:43:15.786829: Yayy! New best EMA pseudo Dice: 0.6933 
+2024-09-04 17:43:24.372845:  
+2024-09-04 17:43:24.374087: Epoch 13 
+2024-09-04 17:43:24.374833: Current learning rate: 0.00988 
+2024-09-04 17:47:40.641185: train_loss -0.5151 
+2024-09-04 17:47:40.656319: val_loss -0.5488 
+2024-09-04 17:47:40.657529: Pseudo dice [0.9124, 0.91, 0.9445, 0.8814, 0.7379, 0.8781, 0.9745] 
+2024-09-04 17:47:40.658350: Epoch time: 256.27 s 
+2024-09-04 17:47:40.659020: Yayy! New best EMA pseudo Dice: 0.7131 
+2024-09-04 17:47:48.938149:  
+2024-09-04 17:47:48.939095: Epoch 14 
+2024-09-04 17:47:48.939863: Current learning rate: 0.00987 
+2024-09-04 17:52:03.865386: train_loss -0.5524 
+2024-09-04 17:52:03.867910: val_loss -0.5943 
+2024-09-04 17:52:03.868679: Pseudo dice [0.9128, 0.9075, 0.947, 0.8765, 0.7372, 0.899, 0.9716] 
+2024-09-04 17:52:03.869416: Epoch time: 254.93 s 
+2024-09-04 17:52:03.870070: Yayy! New best EMA pseudo Dice: 0.7311 
+2024-09-04 17:52:11.898160:  
+2024-09-04 17:52:11.899292: Epoch 15 
+2024-09-04 17:52:11.900279: Current learning rate: 0.00986 
+2024-09-04 17:56:13.048997: train_loss -0.5494 
+2024-09-04 17:56:13.066928: val_loss -0.5767 
+2024-09-04 17:56:13.067949: Pseudo dice [0.9203, 0.9143, 0.9463, 0.8775, 0.7495, 0.9001, 0.9695] 
+2024-09-04 17:56:13.068696: Epoch time: 241.15 s 
+2024-09-04 17:56:13.069655: Yayy! New best EMA pseudo Dice: 0.7477 
+2024-09-04 17:56:21.884744:  
+2024-09-04 17:56:21.885789: Epoch 16 
+2024-09-04 17:56:21.888192: Current learning rate: 0.00986 
+2024-09-04 18:00:43.287884: train_loss -0.5903 
+2024-09-04 18:00:43.289466: val_loss -0.612 
+2024-09-04 18:00:43.290725: Pseudo dice [0.9248, 0.912, 0.9556, 0.903, 0.7764, 0.9024, 0.9766] 
+2024-09-04 18:00:43.291772: Epoch time: 261.41 s 
+2024-09-04 18:00:43.292720: Yayy! New best EMA pseudo Dice: 0.7636 
+2024-09-04 18:00:52.063607:  
+2024-09-04 18:00:52.064583: Epoch 17 
+2024-09-04 18:00:52.065426: Current learning rate: 0.00985 
+2024-09-04 18:05:02.062862: train_loss -0.5814 
+2024-09-04 18:05:02.064780: val_loss -0.6096 
+2024-09-04 18:05:02.065883: Pseudo dice [0.9262, 0.9166, 0.9576, 0.9046, 0.7678, 0.9005, 0.9763] 
+2024-09-04 18:05:02.066792: Epoch time: 250.0 s 
+2024-09-04 18:05:02.067623: Yayy! New best EMA pseudo Dice: 0.778 
+2024-09-04 18:05:16.336205:  
+2024-09-04 18:05:16.337061: Epoch 18 
+2024-09-04 18:05:16.337709: Current learning rate: 0.00984 
+2024-09-04 18:09:27.747186: train_loss -0.5923 
+2024-09-04 18:09:27.754764: val_loss -0.5887 
+2024-09-04 18:09:27.755681: Pseudo dice [0.9249, 0.9157, 0.9558, 0.9107, 0.7609, 0.9035, 0.98] 
+2024-09-04 18:09:27.756390: Epoch time: 251.41 s 
+2024-09-04 18:09:27.757388: Yayy! New best EMA pseudo Dice: 0.7909 
+2024-09-04 18:09:37.600469:  
+2024-09-04 18:09:37.601458: Epoch 19 
+2024-09-04 18:09:37.602204: Current learning rate: 0.00983 
+2024-09-04 18:14:14.771101: train_loss -0.6097 
+2024-09-04 18:14:14.789595: val_loss -0.5657 
+2024-09-04 18:14:14.790488: Pseudo dice [0.9304, 0.9279, 0.9508, 0.8996, 0.7258, 0.8993, 0.9725] 
+2024-09-04 18:14:14.791251: Epoch time: 277.18 s 
+2024-09-04 18:14:14.791991: Yayy! New best EMA pseudo Dice: 0.8019 
+2024-09-04 18:14:24.660793:  
+2024-09-04 18:14:24.663285: Epoch 20 
+2024-09-04 18:14:24.666022: Current learning rate: 0.00982 
+2024-09-04 18:18:23.079908: train_loss -0.605 
+2024-09-04 18:18:23.081358: val_loss -0.6122 
+2024-09-04 18:18:23.082165: Pseudo dice [0.929, 0.9171, 0.9574, 0.8867, 0.7678, 0.913, 0.969] 
+2024-09-04 18:18:23.083967: Epoch time: 238.42 s 
+2024-09-04 18:18:23.084647: Yayy! New best EMA pseudo Dice: 0.8123 
+2024-09-04 18:18:33.888383:  
+2024-09-04 18:18:33.900058: Epoch 21 
+2024-09-04 18:18:34.041418: Current learning rate: 0.00981 
+2024-09-04 18:22:44.389045: train_loss -0.6224 
+2024-09-04 18:22:44.392040: val_loss -0.6319 
+2024-09-04 18:22:44.393360: Pseudo dice [0.9329, 0.9187, 0.9606, 0.9174, 0.7406, 0.9217, 0.9735] 
+2024-09-04 18:22:44.394754: Epoch time: 250.5 s 
+2024-09-04 18:22:44.396650: Yayy! New best EMA pseudo Dice: 0.822 
+2024-09-04 18:22:53.993726:  
+2024-09-04 18:22:53.994720: Epoch 22 
+2024-09-04 18:22:53.995767: Current learning rate: 0.0098 
+2024-09-04 18:27:07.357257: train_loss -0.6214 
+2024-09-04 18:27:07.360274: val_loss -0.6526 
+2024-09-04 18:27:07.361139: Pseudo dice [0.9362, 0.9251, 0.9619, 0.9074, 0.7703, 0.9198, 0.9832] 
+2024-09-04 18:27:07.361850: Epoch time: 253.37 s 
+2024-09-04 18:27:07.362568: Yayy! New best EMA pseudo Dice: 0.8313 
+2024-09-04 18:27:16.544787:  
+2024-09-04 18:27:16.590405: Epoch 23 
+2024-09-04 18:27:16.601921: Current learning rate: 0.00979 
+2024-09-04 18:31:37.326186: train_loss -0.6277 
+2024-09-04 18:31:37.328880: val_loss -0.6639 
+2024-09-04 18:31:37.329648: Pseudo dice [0.9402, 0.9297, 0.9625, 0.9126, 0.7726, 0.9335, 0.9769] 
+2024-09-04 18:31:37.330342: Epoch time: 260.79 s 
+2024-09-04 18:31:37.331239: Yayy! New best EMA pseudo Dice: 0.84 
+2024-09-04 18:31:47.273249:  
+2024-09-04 18:31:47.275213: Epoch 24 
+2024-09-04 18:31:47.278204: Current learning rate: 0.00978 
+2024-09-04 18:35:55.462771: train_loss -0.6357 
+2024-09-04 18:35:55.465489: val_loss -0.6442 
+2024-09-04 18:35:55.466382: Pseudo dice [0.9351, 0.9286, 0.9604, 0.9066, 0.7714, 0.928, 0.9809] 
+2024-09-04 18:35:55.467285: Epoch time: 248.2 s 
+2024-09-04 18:35:55.468005: Yayy! New best EMA pseudo Dice: 0.8476 
+2024-09-04 18:36:04.541625:  
+2024-09-04 18:36:04.542670: Epoch 25 
+2024-09-04 18:36:04.543471: Current learning rate: 0.00977 
+2024-09-04 18:39:59.501812: train_loss -0.6243 
+2024-09-04 18:39:59.504179: val_loss -0.6813 
+2024-09-04 18:39:59.504903: Pseudo dice [0.9375, 0.9254, 0.9611, 0.916, 0.757, 0.9219, 0.9827] 
+2024-09-04 18:39:59.506082: Epoch time: 234.96 s 
+2024-09-04 18:39:59.506964: Yayy! New best EMA pseudo Dice: 0.8543 
+2024-09-04 18:40:07.676234:  
+2024-09-04 18:40:07.677971: Epoch 26 
+2024-09-04 18:40:07.679004: Current learning rate: 0.00977 
+2024-09-04 18:44:04.448398: train_loss -0.6292 
+2024-09-04 18:44:04.451147: val_loss -0.6039 
+2024-09-04 18:44:04.452828: Pseudo dice [0.9322, 0.9258, 0.9569, 0.9112, 0.7978, 0.9239, 0.9796] 
+2024-09-04 18:44:04.454314: Epoch time: 236.78 s 
+2024-09-04 18:44:04.455589: Yayy! New best EMA pseudo Dice: 0.8607 
+2024-09-04 18:44:14.453599:  
+2024-09-04 18:44:14.455084: Epoch 27 
+2024-09-04 18:44:14.455971: Current learning rate: 0.00976 
+2024-09-04 18:48:32.422119: train_loss -0.6293 
+2024-09-04 18:48:32.424381: val_loss -0.6287 
+2024-09-04 18:48:32.426039: Pseudo dice [0.9317, 0.926, 0.9537, 0.9179, 0.8014, 0.9215, 0.9815] 
+2024-09-04 18:48:32.427134: Epoch time: 257.97 s 
+2024-09-04 18:48:32.427767: Yayy! New best EMA pseudo Dice: 0.8665 
+2024-09-04 18:48:40.754309:  
+2024-09-04 18:48:40.855555: Epoch 28 
+2024-09-04 18:48:40.866445: Current learning rate: 0.00975 
+2024-09-04 18:52:53.476957: train_loss -0.6485 
+2024-09-04 18:52:53.551562: val_loss -0.6657 
+2024-09-04 18:52:53.558395: Pseudo dice [0.9419, 0.9349, 0.9623, 0.92, 0.8086, 0.9374, 0.9808] 
+2024-09-04 18:52:53.606143: Epoch time: 252.73 s 
+2024-09-04 18:52:53.618906: Yayy! New best EMA pseudo Dice: 0.8725 
+2024-09-04 18:53:02.249013:  
+2024-09-04 18:53:02.250028: Epoch 29 
+2024-09-04 18:53:02.250717: Current learning rate: 0.00974 
+2024-09-04 18:57:03.227215: train_loss -0.6171 
+2024-09-04 18:57:03.230040: val_loss -0.623 
+2024-09-04 18:57:03.231046: Pseudo dice [0.9373, 0.932, 0.9599, 0.9136, 0.762, 0.9042, 0.9749] 
+2024-09-04 18:57:03.232068: Epoch time: 240.98 s 
+2024-09-04 18:57:03.232776: Yayy! New best EMA pseudo Dice: 0.8765 
+2024-09-04 18:57:12.956476:  
+2024-09-04 18:57:12.958612: Epoch 30 
+2024-09-04 18:57:12.961517: Current learning rate: 0.00973 
+2024-09-04 19:01:37.464990: train_loss -0.6181 
+2024-09-04 19:01:37.486073: val_loss -0.6492 
+2024-09-04 19:01:37.487121: Pseudo dice [0.9405, 0.9321, 0.9599, 0.9168, 0.7682, 0.9328, 0.9836] 
+2024-09-04 19:01:37.487888: Epoch time: 264.51 s 
+2024-09-04 19:01:37.488527: Yayy! New best EMA pseudo Dice: 0.8807 
+2024-09-04 19:01:46.793260:  
+2024-09-04 19:01:46.794797: Epoch 31 
+2024-09-04 19:01:46.795594: Current learning rate: 0.00972 
+2024-09-04 19:06:09.794203: train_loss -0.6302 
+2024-09-04 19:06:09.805069: val_loss -0.6893 
+2024-09-04 19:06:09.806204: Pseudo dice [0.9385, 0.9326, 0.9588, 0.92, 0.799, 0.9344, 0.9806] 
+2024-09-04 19:06:09.807181: Epoch time: 263.01 s 
+2024-09-04 19:06:09.808114: Yayy! New best EMA pseudo Dice: 0.885 
+2024-09-04 19:06:18.786137:  
+2024-09-04 19:06:18.787087: Epoch 32 
+2024-09-04 19:06:18.787892: Current learning rate: 0.00971 
+2024-09-04 19:10:41.697400: train_loss -0.6342 
+2024-09-04 19:10:41.710980: val_loss -0.6297 
+2024-09-04 19:10:41.711752: Pseudo dice [0.9336, 0.933, 0.961, 0.9163, 0.7794, 0.9142, 0.98] 
+2024-09-04 19:10:41.712793: Epoch time: 262.91 s 
+2024-09-04 19:10:41.713409: Yayy! New best EMA pseudo Dice: 0.8882 
+2024-09-04 19:10:51.096732:  
+2024-09-04 19:10:51.097894: Epoch 33 
+2024-09-04 19:10:51.098690: Current learning rate: 0.0097 
+2024-09-04 19:15:06.753737: train_loss -0.6423 
+2024-09-04 19:15:06.756399: val_loss -0.6683 
+2024-09-04 19:15:06.758917: Pseudo dice [0.9412, 0.9318, 0.9633, 0.9211, 0.7961, 0.9236, 0.9848] 
+2024-09-04 19:15:06.760385: Epoch time: 255.66 s 
+2024-09-04 19:15:06.761760: Yayy! New best EMA pseudo Dice: 0.8917 
+2024-09-04 19:15:16.480782:  
+2024-09-04 19:15:16.481841: Epoch 34 
+2024-09-04 19:15:16.482950: Current learning rate: 0.00969 
+2024-09-04 19:19:37.819028: train_loss -0.656 
+2024-09-04 19:19:37.821667: val_loss -0.6215 
+2024-09-04 19:19:37.822444: Pseudo dice [0.9442, 0.9352, 0.9671, 0.9173, 0.7796, 0.9114, 0.9748] 
+2024-09-04 19:19:37.823218: Epoch time: 261.34 s 
+2024-09-04 19:19:37.823923: Yayy! New best EMA pseudo Dice: 0.8943 
+2024-09-04 19:19:46.774284:  
+2024-09-04 19:19:46.775656: Epoch 35 
+2024-09-04 19:19:46.777948: Current learning rate: 0.00968 
+2024-09-04 19:24:07.790254: train_loss -0.6385 
+2024-09-04 19:24:07.792915: val_loss -0.6503 
+2024-09-04 19:24:07.794134: Pseudo dice [0.9387, 0.9311, 0.9616, 0.911, 0.792, 0.9185, 0.9816] 
+2024-09-04 19:24:07.795018: Epoch time: 261.02 s 
+2024-09-04 19:24:07.795667: Yayy! New best EMA pseudo Dice: 0.8968 
+2024-09-04 19:24:16.398162:  
+2024-09-04 19:24:16.399016: Epoch 36 
+2024-09-04 19:24:16.399854: Current learning rate: 0.00968 
+2024-09-04 19:28:34.515062: train_loss -0.6501 
+2024-09-04 19:28:34.518035: val_loss -0.6802 
+2024-09-04 19:28:34.518802: Pseudo dice [0.9435, 0.937, 0.9652, 0.9199, 0.8066, 0.9414, 0.9824] 
+2024-09-04 19:28:34.519521: Epoch time: 258.12 s 
+2024-09-04 19:28:34.520204: Yayy! New best EMA pseudo Dice: 0.9 
+2024-09-04 19:28:43.718831:  
+2024-09-04 19:28:43.719978: Epoch 37 
+2024-09-04 19:28:43.720772: Current learning rate: 0.00967 
+2024-09-04 19:33:01.618123: train_loss -0.6512 
+2024-09-04 19:33:01.658939: val_loss -0.6479 
+2024-09-04 19:33:01.659880: Pseudo dice [0.9451, 0.9364, 0.967, 0.9219, 0.7934, 0.939, 0.9797] 
+2024-09-04 19:33:01.660649: Epoch time: 257.9 s 
+2024-09-04 19:33:01.661304: Yayy! New best EMA pseudo Dice: 0.9026 
+2024-09-04 19:33:10.507086:  
+2024-09-04 19:33:10.507887: Epoch 38 
+2024-09-04 19:33:10.508556: Current learning rate: 0.00966 
+2024-09-04 19:37:26.946207: train_loss -0.6458 
+2024-09-04 19:37:26.947889: val_loss -0.6635 
+2024-09-04 19:37:26.948846: Pseudo dice [0.9427, 0.9369, 0.961, 0.9199, 0.8102, 0.944, 0.9864] 
+2024-09-04 19:37:26.949585: Epoch time: 256.44 s 
+2024-09-04 19:37:26.950158: Yayy! New best EMA pseudo Dice: 0.9052 
+2024-09-04 19:37:35.682974:  
+2024-09-04 19:37:35.683971: Epoch 39 
+2024-09-04 19:37:35.684794: Current learning rate: 0.00965 
+2024-09-04 19:41:43.338387: train_loss -0.6577 
+2024-09-04 19:41:43.341192: val_loss -0.6666 
+2024-09-04 19:41:43.342746: Pseudo dice [0.9439, 0.94, 0.9649, 0.9121, 0.7771, 0.9379, 0.9841] 
+2024-09-04 19:41:43.344664: Epoch time: 247.66 s 
+2024-09-04 19:41:43.345946: Yayy! New best EMA pseudo Dice: 0.9069 
+2024-09-04 19:41:55.238460:  
+2024-09-04 19:41:55.239320: Epoch 40 
+2024-09-04 19:41:55.239933: Current learning rate: 0.00964 
+2024-09-04 19:46:06.800844: train_loss -0.6865 
+2024-09-04 19:46:06.818527: val_loss -0.7004 
+2024-09-04 19:46:06.819841: Pseudo dice [0.9531, 0.9419, 0.9752, 0.9327, 0.8144, 0.9323, 0.9839] 
+2024-09-04 19:46:06.821660: Epoch time: 251.57 s 
+2024-09-04 19:46:06.822718: Yayy! New best EMA pseudo Dice: 0.9096 
+2024-09-04 19:46:16.044251:  
+2024-09-04 19:46:16.045279: Epoch 41 
+2024-09-04 19:46:16.047212: Current learning rate: 0.00963 
+2024-09-04 19:50:21.706962: train_loss -0.6767 
+2024-09-04 19:50:21.710218: val_loss -0.668 
+2024-09-04 19:50:21.711163: Pseudo dice [0.95, 0.9408, 0.9671, 0.9244, 0.7771, 0.934, 0.9842] 
+2024-09-04 19:50:21.712412: Epoch time: 245.67 s 
+2024-09-04 19:50:21.713526: Yayy! New best EMA pseudo Dice: 0.9112 
+2024-09-04 19:50:31.172663:  
+2024-09-04 19:50:31.174095: Epoch 42 
+2024-09-04 19:50:31.175037: Current learning rate: 0.00962 
+2024-09-04 19:54:46.644233: train_loss -0.6689 
+2024-09-04 19:54:46.660393: val_loss -0.6752 
+2024-09-04 19:54:46.661417: Pseudo dice [0.9433, 0.9385, 0.9564, 0.919, 0.7994, 0.9451, 0.9845] 
+2024-09-04 19:54:46.662567: Epoch time: 255.48 s 
+2024-09-04 19:54:46.663424: Yayy! New best EMA pseudo Dice: 0.9127 
+2024-09-04 19:54:54.718816:  
+2024-09-04 19:54:54.719855: Epoch 43 
+2024-09-04 19:54:54.720541: Current learning rate: 0.00961 
+2024-09-04 19:58:57.522810: train_loss -0.6472 
+2024-09-04 19:58:57.524452: val_loss -0.6965 
+2024-09-04 19:58:57.525276: Pseudo dice [0.9457, 0.9381, 0.9648, 0.9187, 0.8189, 0.9458, 0.9853] 
+2024-09-04 19:58:57.526165: Epoch time: 242.81 s 
+2024-09-04 19:58:57.526880: Yayy! New best EMA pseudo Dice: 0.9145 
+2024-09-04 19:59:07.725524:  
+2024-09-04 19:59:07.726561: Epoch 44 
+2024-09-04 19:59:07.727263: Current learning rate: 0.0096 
+2024-09-04 20:03:25.229734: train_loss -0.6725 
+2024-09-04 20:03:25.239110: val_loss -0.677 
+2024-09-04 20:03:25.241648: Pseudo dice [0.942, 0.9351, 0.9655, 0.9248, 0.8198, 0.9365, 0.9877] 
+2024-09-04 20:03:25.244228: Epoch time: 257.51 s 
+2024-09-04 20:03:25.246914: Yayy! New best EMA pseudo Dice: 0.9161 
+2024-09-04 20:03:37.311316:  
+2024-09-04 20:03:37.466535: Epoch 45 
+2024-09-04 20:03:37.574805: Current learning rate: 0.00959 
+2024-09-04 20:07:53.688877: train_loss -0.6473 
+2024-09-04 20:07:53.691976: val_loss -0.6972 
+2024-09-04 20:07:53.692842: Pseudo dice [0.944, 0.936, 0.9656, 0.9253, 0.7937, 0.941, 0.9836] 
+2024-09-04 20:07:53.693656: Epoch time: 256.38 s 
+2024-09-04 20:07:53.694282: Yayy! New best EMA pseudo Dice: 0.9172 
+2024-09-04 20:08:02.211903:  
+2024-09-04 20:08:02.213436: Epoch 46 
+2024-09-04 20:08:02.214867: Current learning rate: 0.00959 
+2024-09-04 20:12:33.740609: train_loss -0.6586 
+2024-09-04 20:12:33.742086: val_loss -0.6832 
+2024-09-04 20:12:33.742806: Pseudo dice [0.9449, 0.936, 0.9675, 0.9312, 0.8349, 0.9448, 0.9879] 
+2024-09-04 20:12:33.743528: Epoch time: 271.53 s 
+2024-09-04 20:12:33.744155: Yayy! New best EMA pseudo Dice: 0.919 
+2024-09-04 20:12:42.120206:  
+2024-09-04 20:12:42.121243: Epoch 47 
+2024-09-04 20:12:42.121995: Current learning rate: 0.00958 
+2024-09-04 20:16:52.592433: train_loss -0.6655 
+2024-09-04 20:16:52.594077: val_loss -0.6736 
+2024-09-04 20:16:52.595061: Pseudo dice [0.9461, 0.9421, 0.9635, 0.9269, 0.8014, 0.9413, 0.9856] 
+2024-09-04 20:16:52.595945: Epoch time: 250.48 s 
+2024-09-04 20:16:52.596655: Yayy! New best EMA pseudo Dice: 0.9201 
+2024-09-04 20:17:01.632829:  
+2024-09-04 20:17:01.633547: Epoch 48 
+2024-09-04 20:17:01.634141: Current learning rate: 0.00957 
+2024-09-04 20:21:32.822239: train_loss -0.6675 
+2024-09-04 20:21:32.839175: val_loss -0.6917 
+2024-09-04 20:21:32.839936: Pseudo dice [0.9503, 0.9397, 0.9715, 0.9283, 0.8012, 0.9275, 0.9839] 
+2024-09-04 20:21:32.840659: Epoch time: 271.19 s 
+2024-09-04 20:21:32.841664: Yayy! New best EMA pseudo Dice: 0.9209 
+2024-09-04 20:21:42.505443:  
+2024-09-04 20:21:42.507249: Epoch 49 
+2024-09-04 20:21:42.508735: Current learning rate: 0.00956 
+2024-09-04 20:25:58.593129: train_loss -0.6793 
+2024-09-04 20:25:58.612925: val_loss -0.6955 
+2024-09-04 20:25:58.614158: Pseudo dice [0.9489, 0.9418, 0.9675, 0.927, 0.8294, 0.9401, 0.9831] 
+2024-09-04 20:25:58.615065: Epoch time: 256.09 s 
+2024-09-04 20:26:06.949714: Yayy! New best EMA pseudo Dice: 0.9222 
+2024-09-04 20:26:17.964293:  
+2024-09-04 20:26:17.965151: Epoch 50 
+2024-09-04 20:26:17.965706: Current learning rate: 0.00955 
+2024-09-04 20:30:34.236341: train_loss -0.6794 
+2024-09-04 20:30:34.251460: val_loss -0.6901 
+2024-09-04 20:30:34.252841: Pseudo dice [0.9526, 0.942, 0.9705, 0.9304, 0.8188, 0.9513, 0.9894] 
+2024-09-04 20:30:34.254277: Epoch time: 256.28 s 
+2024-09-04 20:30:34.255047: Yayy! New best EMA pseudo Dice: 0.9237 
+2024-09-04 20:30:42.215657:  
+2024-09-04 20:30:42.216680: Epoch 51 
+2024-09-04 20:30:42.217365: Current learning rate: 0.00954 
+2024-09-04 20:35:19.022812: train_loss -0.669 
+2024-09-04 20:35:19.024242: val_loss -0.6698 
+2024-09-04 20:35:19.024906: Pseudo dice [0.9471, 0.9418, 0.9672, 0.9283, 0.7997, 0.9412, 0.9866] 
+2024-09-04 20:35:19.025945: Epoch time: 276.81 s 
+2024-09-04 20:35:19.041352: Yayy! New best EMA pseudo Dice: 0.9243 
+2024-09-04 20:35:26.897513:  
+2024-09-04 20:35:26.898768: Epoch 52 
+2024-09-04 20:35:26.899576: Current learning rate: 0.00953 
+2024-09-04 20:39:27.631464: train_loss -0.6907 
+2024-09-04 20:39:27.733220: val_loss -0.6662 
+2024-09-04 20:39:27.741695: Pseudo dice [0.9473, 0.9427, 0.9648, 0.9325, 0.8137, 0.9333, 0.9823] 
+2024-09-04 20:39:27.749815: Epoch time: 240.74 s 
+2024-09-04 20:39:27.810889: Yayy! New best EMA pseudo Dice: 0.925 
+2024-09-04 20:39:37.984676:  
+2024-09-04 20:39:37.985948: Epoch 53 
+2024-09-04 20:39:37.986760: Current learning rate: 0.00952 
+2024-09-04 20:43:32.459882: train_loss -0.6777 
+2024-09-04 20:43:32.471121: val_loss -0.7016 
+2024-09-04 20:43:32.471938: Pseudo dice [0.9508, 0.9443, 0.9691, 0.9369, 0.8222, 0.9503, 0.9874] 
+2024-09-04 20:43:32.472722: Epoch time: 234.48 s 
+2024-09-04 20:43:32.473357: Yayy! New best EMA pseudo Dice: 0.9262 
+2024-09-04 20:43:42.440517:  
+2024-09-04 20:43:42.441837: Epoch 54 
+2024-09-04 20:43:42.442633: Current learning rate: 0.00951 
+2024-09-04 20:48:11.342720: train_loss -0.6703 
+2024-09-04 20:48:11.344958: val_loss -0.6657 
+2024-09-04 20:48:11.345891: Pseudo dice [0.9458, 0.9454, 0.9642, 0.9217, 0.819, 0.9518, 0.9886] 
+2024-09-04 20:48:11.346607: Epoch time: 268.91 s 
+2024-09-04 20:48:11.347269: Yayy! New best EMA pseudo Dice: 0.927 
+2024-09-04 20:48:19.140746:  
+2024-09-04 20:48:19.141625: Epoch 55 
+2024-09-04 20:48:19.142266: Current learning rate: 0.0095 
+2024-09-04 20:52:46.077113: train_loss -0.7044 
+2024-09-04 20:52:46.091200: val_loss -0.7088 
+2024-09-04 20:52:46.092031: Pseudo dice [0.9485, 0.9402, 0.966, 0.9203, 0.82, 0.9513, 0.9895] 
+2024-09-04 20:52:46.092742: Epoch time: 266.94 s 
+2024-09-04 20:52:46.093393: Yayy! New best EMA pseudo Dice: 0.9276 
+2024-09-04 20:52:55.901249:  
+2024-09-04 20:52:55.902239: Epoch 56 
+2024-09-04 20:52:55.902954: Current learning rate: 0.00949 
+2024-09-04 20:57:12.831497: train_loss -0.704 
+2024-09-04 20:57:12.833002: val_loss -0.689 
+2024-09-04 20:57:12.833839: Pseudo dice [0.9492, 0.9466, 0.9699, 0.9324, 0.8027, 0.9509, 0.9843] 
+2024-09-04 20:57:12.834584: Epoch time: 256.93 s 
+2024-09-04 20:57:12.835215: Yayy! New best EMA pseudo Dice: 0.9283 
+2024-09-04 20:57:26.388783:  
+2024-09-04 20:57:26.389708: Epoch 57 
+2024-09-04 20:57:26.390344: Current learning rate: 0.00949 
+2024-09-04 21:01:41.190010: train_loss -0.6955 
+2024-09-04 21:01:41.191450: val_loss -0.6531 
+2024-09-04 21:01:41.192397: Pseudo dice [0.9484, 0.9422, 0.9686, 0.9217, 0.7912, 0.9088, 0.9786] 
+2024-09-04 21:01:41.193324: Epoch time: 254.8 s 
+2024-09-04 21:01:43.339845:  
+2024-09-04 21:01:43.340929: Epoch 58 
+2024-09-04 21:01:43.341763: Current learning rate: 0.00948 
+2024-09-04 21:06:08.824463: train_loss -0.675 
+2024-09-04 21:06:08.836200: val_loss -0.7156 
+2024-09-04 21:06:08.837057: Pseudo dice [0.948, 0.936, 0.9727, 0.9277, 0.8174, 0.9298, 0.9853] 
+2024-09-04 21:06:08.837882: Epoch time: 265.49 s 
+2024-09-04 21:06:11.035717:  
+2024-09-04 21:06:11.036651: Epoch 59 
+2024-09-04 21:06:11.037422: Current learning rate: 0.00947 
+2024-09-04 21:10:37.520109: train_loss -0.688 
+2024-09-04 21:10:37.521582: val_loss -0.6907 
+2024-09-04 21:10:37.522302: Pseudo dice [0.9481, 0.9432, 0.9652, 0.928, 0.8228, 0.9486, 0.9884] 
+2024-09-04 21:10:37.523071: Epoch time: 266.49 s 
+2024-09-04 21:10:37.523660: Yayy! New best EMA pseudo Dice: 0.9287 
+2024-09-04 21:10:47.010734:  
+2024-09-04 21:10:47.012194: Epoch 60 
+2024-09-04 21:10:47.014413: Current learning rate: 0.00946 
+2024-09-04 21:14:53.736247: train_loss -0.6864 
+2024-09-04 21:14:53.743622: val_loss -0.6458 
+2024-09-04 21:14:53.744584: Pseudo dice [0.954, 0.9478, 0.9734, 0.9284, 0.7667, 0.9321, 0.9809] 
+2024-09-04 21:14:53.745387: Epoch time: 246.73 s 
+2024-09-04 21:14:56.609351:  
+2024-09-04 21:14:56.610416: Epoch 61 
+2024-09-04 21:14:56.611130: Current learning rate: 0.00945 
+2024-09-04 21:19:02.646597: train_loss -0.6722 
+2024-09-04 21:19:02.658417: val_loss -0.7036 
+2024-09-04 21:19:02.659390: Pseudo dice [0.9543, 0.9474, 0.9716, 0.9339, 0.8396, 0.9494, 0.991] 
+2024-09-04 21:19:02.660182: Epoch time: 246.04 s 
+2024-09-04 21:19:02.660850: Yayy! New best EMA pseudo Dice: 0.9297 
+2024-09-04 21:19:13.775784:  
+2024-09-04 21:19:13.776693: Epoch 62 
+2024-09-04 21:19:13.777271: Current learning rate: 0.00944 
+2024-09-04 21:23:28.925298: train_loss -0.6911 
+2024-09-04 21:23:28.932149: val_loss -0.7186 
+2024-09-04 21:23:28.932989: Pseudo dice [0.9544, 0.9471, 0.9718, 0.9289, 0.8372, 0.9507, 0.99] 
+2024-09-04 21:23:28.934500: Epoch time: 255.15 s 
+2024-09-04 21:23:28.935168: Yayy! New best EMA pseudo Dice: 0.9308 
+2024-09-04 21:23:37.673012:  
+2024-09-04 21:23:37.674298: Epoch 63 
+2024-09-04 21:23:37.675288: Current learning rate: 0.00943 
+2024-09-04 21:27:46.258311: train_loss -0.6864 
+2024-09-04 21:27:46.259780: val_loss -0.7165 
+2024-09-04 21:27:46.260565: Pseudo dice [0.9471, 0.9414, 0.9678, 0.9294, 0.8148, 0.9524, 0.9862] 
+2024-09-04 21:27:46.261404: Epoch time: 248.59 s 
+2024-09-04 21:27:46.262372: Yayy! New best EMA pseudo Dice: 0.9311 
+2024-09-04 21:27:53.473321:  
+2024-09-04 21:27:53.474534: Epoch 64 
+2024-09-04 21:27:53.475393: Current learning rate: 0.00942 
+2024-09-04 21:32:12.151477: train_loss -0.706 
+2024-09-04 21:32:12.152819: val_loss -0.6865 
+2024-09-04 21:32:12.153624: Pseudo dice [0.9501, 0.9412, 0.9665, 0.9367, 0.826, 0.9389, 0.9829] 
+2024-09-04 21:32:12.154407: Epoch time: 258.68 s 
+2024-09-04 21:32:12.155207: Yayy! New best EMA pseudo Dice: 0.9314 
+2024-09-04 21:32:21.244930:  
+2024-09-04 21:32:21.246037: Epoch 65 
+2024-09-04 21:32:21.246931: Current learning rate: 0.00941 
+2024-09-04 21:36:36.963862: train_loss -0.6838 
+2024-09-04 21:36:36.965444: val_loss -0.6795 
+2024-09-04 21:36:36.966430: Pseudo dice [0.9538, 0.9455, 0.9704, 0.9322, 0.8117, 0.9284, 0.9803] 
+2024-09-04 21:36:36.967271: Epoch time: 255.72 s 
+2024-09-04 21:36:36.967930: Yayy! New best EMA pseudo Dice: 0.9315 
+2024-09-04 21:36:50.383324:  
+2024-09-04 21:36:50.384266: Epoch 66 
+2024-09-04 21:36:50.385087: Current learning rate: 0.0094 
+2024-09-04 21:40:59.889405: train_loss -0.6959 
+2024-09-04 21:40:59.902677: val_loss -0.6915 
+2024-09-04 21:40:59.903573: Pseudo dice [0.9548, 0.9443, 0.9726, 0.9332, 0.829, 0.9484, 0.9873] 
+2024-09-04 21:40:59.904420: Epoch time: 249.51 s 
+2024-09-04 21:40:59.905333: Yayy! New best EMA pseudo Dice: 0.9322 
+2024-09-04 21:41:09.037478:  
+2024-09-04 21:41:09.038809: Epoch 67 
+2024-09-04 21:41:09.040017: Current learning rate: 0.00939 
+2024-09-04 21:45:23.267352: train_loss -0.701 
+2024-09-04 21:45:23.268747: val_loss -0.7187 
+2024-09-04 21:45:23.269557: Pseudo dice [0.953, 0.947, 0.9703, 0.9323, 0.8322, 0.9552, 0.9861] 
+2024-09-04 21:45:23.270270: Epoch time: 254.23 s 
+2024-09-04 21:45:23.270936: Yayy! New best EMA pseudo Dice: 0.9329 
+2024-09-04 21:45:33.000012:  
+2024-09-04 21:45:33.001405: Epoch 68 
+2024-09-04 21:45:33.002290: Current learning rate: 0.00939 
+2024-09-04 21:50:11.729523: train_loss -0.6774 
+2024-09-04 21:50:11.739674: val_loss -0.6971 
+2024-09-04 21:50:11.740541: Pseudo dice [0.9492, 0.9414, 0.9656, 0.9325, 0.8177, 0.9511, 0.9845] 
+2024-09-04 21:50:11.741287: Epoch time: 278.73 s 
+2024-09-04 21:50:11.741939: Yayy! New best EMA pseudo Dice: 0.9331 
+2024-09-04 21:50:21.674184:  
+2024-09-04 21:50:21.675714: Epoch 69 
+2024-09-04 21:50:21.676581: Current learning rate: 0.00938 
+2024-09-04 21:54:36.164093: train_loss -0.6864 
+2024-09-04 21:54:36.166407: val_loss -0.6702 
+2024-09-04 21:54:36.167382: Pseudo dice [0.9506, 0.9447, 0.9638, 0.9266, 0.8358, 0.9527, 0.9871] 
+2024-09-04 21:54:36.168271: Epoch time: 254.49 s 
+2024-09-04 21:54:36.169044: Yayy! New best EMA pseudo Dice: 0.9335 
+2024-09-04 21:54:45.090587:  
+2024-09-04 21:54:45.094315: Epoch 70 
+2024-09-04 21:54:45.095773: Current learning rate: 0.00937 
+2024-09-04 21:59:05.596321: train_loss -0.6843 
+2024-09-04 21:59:05.599839: val_loss -0.6776 
+2024-09-04 21:59:05.601965: Pseudo dice [0.9502, 0.9455, 0.9697, 0.9297, 0.789, 0.9357, 0.9832] 
+2024-09-04 21:59:05.603722: Epoch time: 260.51 s 
+2024-09-04 21:59:08.222161:  
+2024-09-04 21:59:08.223454: Epoch 71 
+2024-09-04 21:59:08.225039: Current learning rate: 0.00936 
+2024-09-04 22:03:25.444475: train_loss -0.6773 
+2024-09-04 22:03:25.461291: val_loss -0.7034 
+2024-09-04 22:03:25.463233: Pseudo dice [0.9506, 0.9461, 0.9728, 0.9264, 0.7816, 0.9436, 0.9829] 
+2024-09-04 22:03:25.464550: Epoch time: 257.23 s 
+2024-09-04 22:03:28.286146:  
+2024-09-04 22:03:28.287042: Epoch 72 
+2024-09-04 22:03:28.288044: Current learning rate: 0.00935 
+2024-09-04 22:07:56.589908: train_loss -0.676 
+2024-09-04 22:07:56.592748: val_loss -0.7106 
+2024-09-04 22:07:56.594192: Pseudo dice [0.9498, 0.9443, 0.9679, 0.9352, 0.8363, 0.949, 0.9881] 
+2024-09-04 22:07:56.594989: Epoch time: 268.31 s 
+2024-09-04 22:07:59.016798:  
+2024-09-04 22:07:59.018361: Epoch 73 
+2024-09-04 22:07:59.019387: Current learning rate: 0.00934 
+2024-09-04 22:12:10.962900: train_loss -0.6944 
+2024-09-04 22:12:10.988453: val_loss -0.7345 
+2024-09-04 22:12:10.989335: Pseudo dice [0.953, 0.9464, 0.9688, 0.9312, 0.8376, 0.9547, 0.9861] 
+2024-09-04 22:12:10.990126: Epoch time: 251.95 s 
+2024-09-04 22:12:10.990777: Yayy! New best EMA pseudo Dice: 0.9339 
+2024-09-04 22:12:20.340696:  
+2024-09-04 22:12:20.342320: Epoch 74 
+2024-09-04 22:12:20.343804: Current learning rate: 0.00933 
+2024-09-04 22:16:37.127758: train_loss -0.7028 
+2024-09-04 22:16:37.139318: val_loss -0.6601 
+2024-09-04 22:16:37.140315: Pseudo dice [0.9559, 0.9491, 0.972, 0.9351, 0.7929, 0.9582, 0.9858] 
+2024-09-04 22:16:37.141051: Epoch time: 256.79 s 
+2024-09-04 22:16:37.141732: Yayy! New best EMA pseudo Dice: 0.9341 
+2024-09-04 22:16:45.366501:  
+2024-09-04 22:16:45.367412: Epoch 75 
+2024-09-04 22:16:45.368114: Current learning rate: 0.00932 
+2024-09-04 22:20:53.724226: train_loss -0.6911 
+2024-09-04 22:20:53.743236: val_loss -0.7022 
+2024-09-04 22:20:53.744314: Pseudo dice [0.9521, 0.9447, 0.9685, 0.9276, 0.8, 0.9551, 0.9854] 
+2024-09-04 22:20:53.745087: Epoch time: 248.36 s 
+2024-09-04 22:20:56.392700:  
+2024-09-04 22:20:56.393583: Epoch 76 
+2024-09-04 22:20:56.394276: Current learning rate: 0.00931 
+2024-09-04 22:25:09.910873: train_loss -0.6568 
+2024-09-04 22:25:09.912542: val_loss -0.6821 
+2024-09-04 22:25:09.913388: Pseudo dice [0.9506, 0.9425, 0.9661, 0.9287, 0.805, 0.9522, 0.9867] 
+2024-09-04 22:25:09.914093: Epoch time: 253.52 s 
+2024-09-04 22:25:14.846961:  
+2024-09-04 22:25:14.848009: Epoch 77 
+2024-09-04 22:25:14.848976: Current learning rate: 0.0093 
+2024-09-04 22:29:20.263861: train_loss -0.6912 
+2024-09-04 22:29:20.267976: val_loss -0.6904 
+2024-09-04 22:29:20.268896: Pseudo dice [0.9474, 0.9447, 0.9629, 0.9321, 0.8318, 0.9363, 0.9861] 
+2024-09-04 22:29:20.269670: Epoch time: 245.42 s 
+2024-09-04 22:29:22.740394:  
+2024-09-04 22:29:22.741494: Epoch 78 
+2024-09-04 22:29:22.742417: Current learning rate: 0.0093 
+2024-09-04 22:33:35.598497: train_loss -0.6921 
+2024-09-04 22:33:35.600885: val_loss -0.69 
+2024-09-04 22:33:35.601596: Pseudo dice [0.9543, 0.9479, 0.9727, 0.9365, 0.8347, 0.9516, 0.9782] 
+2024-09-04 22:33:35.609249: Epoch time: 252.86 s 
+2024-09-04 22:33:35.609932: Yayy! New best EMA pseudo Dice: 0.9345 
+2024-09-04 22:33:45.005116:  
+2024-09-04 22:33:45.006168: Epoch 79 
+2024-09-04 22:33:45.006907: Current learning rate: 0.00929 
+2024-09-04 22:38:10.332013: train_loss -0.6818 
+2024-09-04 22:38:10.335527: val_loss -0.6838 
+2024-09-04 22:38:10.338245: Pseudo dice [0.951, 0.9462, 0.9722, 0.934, 0.7909, 0.9509, 0.9818] 
+2024-09-04 22:38:10.340152: Epoch time: 265.33 s 
+2024-09-04 22:38:13.075952:  
+2024-09-04 22:38:13.076941: Epoch 80 
+2024-09-04 22:38:13.077773: Current learning rate: 0.00928 
+2024-09-04 22:42:35.059345: train_loss -0.6848 
+2024-09-04 22:42:35.076002: val_loss -0.7061 
+2024-09-04 22:42:35.076787: Pseudo dice [0.9544, 0.949, 0.9742, 0.9303, 0.8185, 0.9512, 0.9854] 
+2024-09-04 22:42:35.077506: Epoch time: 261.99 s 
+2024-09-04 22:42:35.078262: Yayy! New best EMA pseudo Dice: 0.9346 
+2024-09-04 22:42:44.420386:  
+2024-09-04 22:42:44.421973: Epoch 81 
+2024-09-04 22:42:44.423437: Current learning rate: 0.00927 
+2024-09-04 22:47:01.176721: train_loss -0.7011 
+2024-09-04 22:47:01.179547: val_loss -0.6666 
+2024-09-04 22:47:01.180347: Pseudo dice [0.9513, 0.9453, 0.9705, 0.9244, 0.8011, 0.9425, 0.9823] 
+2024-09-04 22:47:01.181156: Epoch time: 256.76 s 
+2024-09-04 22:47:03.683347:  
+2024-09-04 22:47:03.684167: Epoch 82 
+2024-09-04 22:47:03.684864: Current learning rate: 0.00926 
+2024-09-04 22:51:15.681484: train_loss -0.6872 
+2024-09-04 22:51:15.703704: val_loss -0.7227 
+2024-09-04 22:51:15.704460: Pseudo dice [0.9531, 0.9463, 0.9678, 0.9338, 0.8287, 0.9367, 0.9872] 
+2024-09-04 22:51:15.705162: Epoch time: 252.0 s 
+2024-09-04 22:51:18.255170:  
+2024-09-04 22:51:18.256083: Epoch 83 
+2024-09-04 22:51:18.257018: Current learning rate: 0.00925 
+2024-09-04 22:55:37.510646: train_loss -0.6902 
+2024-09-04 22:55:37.514484: val_loss -0.721 
+2024-09-04 22:55:37.515754: Pseudo dice [0.9533, 0.9462, 0.9725, 0.9318, 0.8281, 0.9327, 0.9839] 
+2024-09-04 22:55:37.517020: Epoch time: 259.26 s 
+2024-09-04 22:55:39.732846:  
+2024-09-04 22:55:39.733781: Epoch 84 
+2024-09-04 22:55:39.734411: Current learning rate: 0.00924 
+2024-09-04 23:00:02.192448: train_loss -0.689 
+2024-09-04 23:00:02.194084: val_loss -0.7155 
+2024-09-04 23:00:02.194777: Pseudo dice [0.9503, 0.9384, 0.9702, 0.9234, 0.838, 0.9549, 0.9893] 
+2024-09-04 23:00:02.195663: Epoch time: 262.46 s 
+2024-09-04 23:00:02.196831: Yayy! New best EMA pseudo Dice: 0.9349 
+2024-09-04 23:00:11.753716:  
+2024-09-04 23:00:11.754668: Epoch 85 
+2024-09-04 23:00:11.755314: Current learning rate: 0.00923 
+2024-09-04 23:04:33.416870: train_loss -0.7044 
+2024-09-04 23:04:33.418496: val_loss -0.6906 
+2024-09-04 23:04:33.419470: Pseudo dice [0.9498, 0.9424, 0.9666, 0.9324, 0.8282, 0.9535, 0.9872] 
+2024-09-04 23:04:33.420400: Epoch time: 261.67 s 
+2024-09-04 23:04:33.421311: Yayy! New best EMA pseudo Dice: 0.9351 
+2024-09-04 23:04:42.750010:  
+2024-09-04 23:04:42.750917: Epoch 86 
+2024-09-04 23:04:42.751633: Current learning rate: 0.00922 
+2024-09-04 23:08:49.558491: train_loss -0.696 
+2024-09-04 23:08:49.567885: val_loss -0.7148 
+2024-09-04 23:08:49.569456: Pseudo dice [0.9527, 0.9464, 0.9709, 0.9341, 0.8336, 0.9546, 0.9887] 
+2024-09-04 23:08:49.570698: Epoch time: 246.81 s 
+2024-09-04 23:08:49.571380: Yayy! New best EMA pseudo Dice: 0.9356 
+2024-09-04 23:08:58.792642:  
+2024-09-04 23:08:58.793629: Epoch 87 
+2024-09-04 23:08:58.794290: Current learning rate: 0.00921 
+2024-09-04 23:13:17.238262: train_loss -0.707 
+2024-09-04 23:13:17.241642: val_loss -0.7256 
+2024-09-04 23:13:17.242425: Pseudo dice [0.9553, 0.9489, 0.9714, 0.9323, 0.8442, 0.9482, 0.9871] 
+2024-09-04 23:13:17.243218: Epoch time: 258.45 s 
+2024-09-04 23:13:17.244066: Yayy! New best EMA pseudo Dice: 0.9362 
+2024-09-04 23:13:26.514562:  
+2024-09-04 23:13:26.515552: Epoch 88 
+2024-09-04 23:13:26.516300: Current learning rate: 0.0092 
+2024-09-04 23:17:48.322264: train_loss -0.689 
+2024-09-04 23:17:48.336895: val_loss -0.6899 
+2024-09-04 23:17:48.337962: Pseudo dice [0.9529, 0.9452, 0.9683, 0.9362, 0.8259, 0.9522, 0.986] 
+2024-09-04 23:17:48.338811: Epoch time: 261.81 s 
+2024-09-04 23:17:48.339536: Yayy! New best EMA pseudo Dice: 0.9364 
+2024-09-04 23:17:57.263329:  
+2024-09-04 23:17:57.264475: Epoch 89 
+2024-09-04 23:17:57.265422: Current learning rate: 0.0092 
+2024-09-04 23:22:14.083997: train_loss -0.6797 
+2024-09-04 23:22:14.086270: val_loss -0.6987 
+2024-09-04 23:22:14.087468: Pseudo dice [0.9543, 0.948, 0.9716, 0.9389, 0.8432, 0.9404, 0.9844] 
+2024-09-04 23:22:14.088573: Epoch time: 256.83 s 
+2024-09-04 23:22:14.089653: Yayy! New best EMA pseudo Dice: 0.9367 
+2024-09-04 23:22:26.514561:  
+2024-09-04 23:22:26.515484: Epoch 90 
+2024-09-04 23:22:26.516211: Current learning rate: 0.00919 
+2024-09-04 23:26:25.550919: train_loss -0.693 
+2024-09-04 23:26:25.563319: val_loss -0.7071 
+2024-09-04 23:26:25.564263: Pseudo dice [0.9545, 0.9489, 0.9701, 0.9291, 0.7965, 0.957, 0.985] 
+2024-09-04 23:26:25.565158: Epoch time: 239.04 s 
+2024-09-04 23:26:27.772421:  
+2024-09-04 23:26:27.773525: Epoch 91 
+2024-09-04 23:26:27.774289: Current learning rate: 0.00918 
+2024-09-04 23:30:50.780244: train_loss -0.7042 
+2024-09-04 23:30:50.795738: val_loss -0.699 
+2024-09-04 23:30:50.796919: Pseudo dice [0.9564, 0.9462, 0.9746, 0.9258, 0.8287, 0.9313, 0.9829] 
+2024-09-04 23:30:50.798089: Epoch time: 263.01 s 
+2024-09-04 23:30:53.175341:  
+2024-09-04 23:30:53.176527: Epoch 92 
+2024-09-04 23:30:53.177350: Current learning rate: 0.00917 
+2024-09-04 23:35:11.173343: train_loss -0.7094 
+2024-09-04 23:35:11.184299: val_loss -0.6859 
+2024-09-04 23:35:11.185154: Pseudo dice [0.9554, 0.9475, 0.9701, 0.9249, 0.8158, 0.9495, 0.9847] 
+2024-09-04 23:35:11.186014: Epoch time: 258.0 s 
+2024-09-04 23:35:13.362675:  
+2024-09-04 23:35:13.363671: Epoch 93 
+2024-09-04 23:35:13.364440: Current learning rate: 0.00916 
+2024-09-04 23:39:31.612030: train_loss -0.7117 
+2024-09-04 23:39:31.628175: val_loss -0.6929 
+2024-09-04 23:39:31.629158: Pseudo dice [0.9538, 0.948, 0.9732, 0.9325, 0.8355, 0.9559, 0.985] 
+2024-09-04 23:39:31.630069: Epoch time: 258.25 s 
+2024-09-04 23:39:34.078006:  
+2024-09-04 23:39:34.079369: Epoch 94 
+2024-09-04 23:39:34.080454: Current learning rate: 0.00915 
+2024-09-04 23:44:14.852712: train_loss -0.6931 
+2024-09-04 23:44:14.883322: val_loss -0.7102 
+2024-09-04 23:44:14.884426: Pseudo dice [0.9539, 0.9478, 0.9674, 0.9337, 0.8489, 0.9553, 0.9878] 
+2024-09-04 23:44:14.885260: Epoch time: 280.78 s 
+2024-09-04 23:44:14.886099: Yayy! New best EMA pseudo Dice: 0.9372 
+2024-09-04 23:44:24.853061:  
+2024-09-04 23:44:24.854070: Epoch 95 
+2024-09-04 23:44:24.854775: Current learning rate: 0.00914 
+2024-09-04 23:48:36.961980: train_loss -0.6912 
+2024-09-04 23:48:36.964269: val_loss -0.6552 
+2024-09-04 23:48:36.965169: Pseudo dice [0.9518, 0.9427, 0.9692, 0.9308, 0.8219, 0.9331, 0.9852] 
+2024-09-04 23:48:36.965952: Epoch time: 252.11 s 
+2024-09-04 23:48:39.103974:  
+2024-09-04 23:48:39.104932: Epoch 96 
+2024-09-04 23:48:39.105648: Current learning rate: 0.00913 
+2024-09-04 23:53:00.925603: train_loss -0.7007 
+2024-09-04 23:53:00.927022: val_loss -0.7328 
+2024-09-04 23:53:00.927879: Pseudo dice [0.9527, 0.9447, 0.9727, 0.9376, 0.8356, 0.9574, 0.9896] 
+2024-09-04 23:53:00.928501: Epoch time: 261.83 s 
+2024-09-04 23:53:00.929124: Yayy! New best EMA pseudo Dice: 0.9373 
+2024-09-04 23:53:09.734805:  
+2024-09-04 23:53:09.735590: Epoch 97 
+2024-09-04 23:53:09.736139: Current learning rate: 0.00912 
+2024-09-04 23:57:16.504366: train_loss -0.6877 
+2024-09-04 23:57:16.515740: val_loss -0.7018 
+2024-09-04 23:57:16.516662: Pseudo dice [0.9546, 0.9484, 0.9698, 0.9306, 0.8384, 0.9427, 0.9882] 
+2024-09-04 23:57:16.517486: Epoch time: 246.77 s 
+2024-09-04 23:57:16.518301: Yayy! New best EMA pseudo Dice: 0.9375 
+2024-09-04 23:57:24.624479:  
+2024-09-04 23:57:24.625303: Epoch 98 
+2024-09-04 23:57:24.625942: Current learning rate: 0.00911 
+2024-09-05 00:01:28.335772: train_loss -0.6988 
+2024-09-05 00:01:28.365686: val_loss -0.7266 
+2024-09-05 00:01:28.366526: Pseudo dice [0.9551, 0.947, 0.972, 0.9361, 0.8126, 0.9553, 0.9849] 
+2024-09-05 00:01:28.367469: Epoch time: 243.71 s 
+2024-09-05 00:01:28.368316: Yayy! New best EMA pseudo Dice: 0.9375 
+2024-09-05 00:01:38.501884:  
+2024-09-05 00:01:38.550622: Epoch 99 
+2024-09-05 00:01:38.613036: Current learning rate: 0.0091 
+2024-09-05 00:05:48.422024: train_loss -0.6953 
+2024-09-05 00:05:48.423505: val_loss -0.685 
+2024-09-05 00:05:48.424504: Pseudo dice [0.9539, 0.9485, 0.9736, 0.9335, 0.8344, 0.9557, 0.9854] 
+2024-09-05 00:05:48.426402: Epoch time: 249.92 s 
+2024-09-05 00:05:55.095816: Yayy! New best EMA pseudo Dice: 0.9378 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_5_00_32_01.txt

@@ -0,0 +1,751 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-05 00:32:04.933750: Using torch.compile... 
+2024-09-05 00:32:14.345654: do_dummy_2d_data_aug: False 
+2024-09-05 00:32:14.352063: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-05 00:32:14.354359: The split file contains 5 splits. 
+2024-09-05 00:32:14.354964: Desired fold for training: 1 
+2024-09-05 00:32:14.355574: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-05 00:32:24.401877: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-05 00:32:28.210971:  
+2024-09-05 00:32:28.223851: Epoch 100 
+2024-09-05 00:32:28.226165: Current learning rate: 0.0091 
+2024-09-05 00:50:39.088884: train_loss -0.6994 
+2024-09-05 00:50:39.113271: val_loss -0.6814 
+2024-09-05 00:50:39.113806: Pseudo dice [0.9572, 0.9491, 0.9729, 0.9364, 0.7831, 0.9509, 0.9849] 
+2024-09-05 00:50:39.114477: Epoch time: 1090.88 s 
+2024-09-05 00:50:41.232970:  
+2024-09-05 00:50:41.234310: Epoch 101 
+2024-09-05 00:50:41.235425: Current learning rate: 0.00909 
+2024-09-05 00:54:53.940321: train_loss -0.6976 
+2024-09-05 00:54:53.956495: val_loss -0.6821 
+2024-09-05 00:54:53.957441: Pseudo dice [0.9523, 0.9485, 0.9679, 0.9337, 0.8274, 0.9419, 0.9863] 
+2024-09-05 00:54:53.959132: Epoch time: 252.71 s 
+2024-09-05 00:54:56.859149:  
+2024-09-05 00:54:56.860163: Epoch 102 
+2024-09-05 00:54:56.860947: Current learning rate: 0.00908 
+2024-09-05 00:59:02.167294: train_loss -0.7047 
+2024-09-05 00:59:02.175528: val_loss -0.7078 
+2024-09-05 00:59:02.176631: Pseudo dice [0.9557, 0.9493, 0.9711, 0.9319, 0.8392, 0.9594, 0.9866] 
+2024-09-05 00:59:02.177516: Epoch time: 245.31 s 
+2024-09-05 00:59:02.178564: Yayy! New best EMA pseudo Dice: 0.9378 
+2024-09-05 00:59:10.470510:  
+2024-09-05 00:59:10.471866: Epoch 103 
+2024-09-05 00:59:10.473238: Current learning rate: 0.00907 
+2024-09-05 01:03:09.841889: train_loss -0.7014 
+2024-09-05 01:03:09.857424: val_loss -0.7023 
+2024-09-05 01:03:09.858269: Pseudo dice [0.9533, 0.9434, 0.97, 0.9287, 0.8375, 0.9504, 0.9871] 
+2024-09-05 01:03:09.859127: Epoch time: 239.38 s 
+2024-09-05 01:03:09.859794: Yayy! New best EMA pseudo Dice: 0.9379 
+2024-09-05 01:03:19.886041:  
+2024-09-05 01:03:19.887090: Epoch 104 
+2024-09-05 01:03:19.887753: Current learning rate: 0.00906 
+2024-09-05 01:07:43.770973: train_loss -0.6861 
+2024-09-05 01:07:43.864116: val_loss -0.7114 
+2024-09-05 01:07:43.993143: Pseudo dice [0.954, 0.9464, 0.9701, 0.9333, 0.8221, 0.9478, 0.9821] 
+2024-09-05 01:07:44.049083: Epoch time: 263.89 s 
+2024-09-05 01:07:46.810663:  
+2024-09-05 01:07:46.826787: Epoch 105 
+2024-09-05 01:07:46.875192: Current learning rate: 0.00905 
+2024-09-05 01:12:25.618450: train_loss -0.7111 
+2024-09-05 01:12:25.634814: val_loss -0.6876 
+2024-09-05 01:12:25.635790: Pseudo dice [0.9528, 0.95, 0.9753, 0.9302, 0.7908, 0.9536, 0.9879] 
+2024-09-05 01:12:25.636536: Epoch time: 278.81 s 
+2024-09-05 01:12:28.245269:  
+2024-09-05 01:12:28.246374: Epoch 106 
+2024-09-05 01:12:28.247085: Current learning rate: 0.00904 
+2024-09-05 01:16:55.594903: train_loss -0.7133 
+2024-09-05 01:16:55.596123: val_loss -0.723 
+2024-09-05 01:16:55.596798: Pseudo dice [0.9569, 0.9469, 0.976, 0.9301, 0.8533, 0.9576, 0.9904] 
+2024-09-05 01:16:55.597370: Epoch time: 267.36 s 
+2024-09-05 01:16:55.597887: Yayy! New best EMA pseudo Dice: 0.9381 
+2024-09-05 01:17:05.232583:  
+2024-09-05 01:17:05.233516: Epoch 107 
+2024-09-05 01:17:05.234137: Current learning rate: 0.00903 
+2024-09-05 01:21:24.819471: train_loss -0.7046 
+2024-09-05 01:21:24.835108: val_loss -0.6741 
+2024-09-05 01:21:24.836040: Pseudo dice [0.9474, 0.9448, 0.9639, 0.9285, 0.8065, 0.9559, 0.9849] 
+2024-09-05 01:21:24.836756: Epoch time: 259.59 s 
+2024-09-05 01:21:27.297509:  
+2024-09-05 01:21:27.298625: Epoch 108 
+2024-09-05 01:21:27.299443: Current learning rate: 0.00902 
+2024-09-05 01:25:38.995745: train_loss -0.6691 
+2024-09-05 01:25:38.998658: val_loss -0.6918 
+2024-09-05 01:25:39.000405: Pseudo dice [0.9513, 0.9468, 0.9677, 0.9318, 0.8578, 0.9443, 0.9834] 
+2024-09-05 01:25:39.003280: Epoch time: 251.7 s 
+2024-09-05 01:25:41.329351:  
+2024-09-05 01:25:41.330672: Epoch 109 
+2024-09-05 01:25:41.331681: Current learning rate: 0.00901 
+2024-09-05 01:30:02.676511: train_loss -0.6952 
+2024-09-05 01:30:02.679241: val_loss -0.6843 
+2024-09-05 01:30:02.680009: Pseudo dice [0.9457, 0.9454, 0.97, 0.9307, 0.8256, 0.9511, 0.9874] 
+2024-09-05 01:30:02.680646: Epoch time: 261.35 s 
+2024-09-05 01:30:04.813842:  
+2024-09-05 01:30:04.815108: Epoch 110 
+2024-09-05 01:30:04.815978: Current learning rate: 0.009 
+2024-09-05 01:34:17.722495: train_loss -0.688 
+2024-09-05 01:34:17.726942: val_loss -0.7412 
+2024-09-05 01:34:17.727903: Pseudo dice [0.954, 0.9461, 0.9704, 0.9367, 0.845, 0.9553, 0.985] 
+2024-09-05 01:34:17.728837: Epoch time: 252.91 s 
+2024-09-05 01:34:17.729506: Yayy! New best EMA pseudo Dice: 0.9382 
+2024-09-05 01:34:27.096425:  
+2024-09-05 01:34:27.098069: Epoch 111 
+2024-09-05 01:34:27.099911: Current learning rate: 0.009 
+2024-09-05 01:38:40.992336: train_loss -0.6849 
+2024-09-05 01:38:41.036831: val_loss -0.7131 
+2024-09-05 01:38:41.038125: Pseudo dice [0.9513, 0.9456, 0.967, 0.928, 0.8275, 0.9399, 0.982] 
+2024-09-05 01:38:41.039319: Epoch time: 253.9 s 
+2024-09-05 01:38:43.559494:  
+2024-09-05 01:38:43.560437: Epoch 112 
+2024-09-05 01:38:43.561154: Current learning rate: 0.00899 
+2024-09-05 01:42:59.056451: train_loss -0.6805 
+2024-09-05 01:42:59.087975: val_loss -0.7228 
+2024-09-05 01:42:59.089586: Pseudo dice [0.9543, 0.9461, 0.9733, 0.9388, 0.8484, 0.9327, 0.9834] 
+2024-09-05 01:42:59.092656: Epoch time: 255.5 s 
+2024-09-05 01:43:01.699583:  
+2024-09-05 01:43:01.700642: Epoch 113 
+2024-09-05 01:43:01.701372: Current learning rate: 0.00898 
+2024-09-05 01:47:20.057437: train_loss -0.6944 
+2024-09-05 01:47:20.074954: val_loss -0.7355 
+2024-09-05 01:47:20.075950: Pseudo dice [0.9577, 0.9496, 0.9739, 0.9312, 0.8366, 0.9525, 0.9853] 
+2024-09-05 01:47:20.076590: Epoch time: 258.36 s 
+2024-09-05 01:47:20.077032: Yayy! New best EMA pseudo Dice: 0.9383 
+2024-09-05 01:47:28.431579:  
+2024-09-05 01:47:28.432450: Epoch 114 
+2024-09-05 01:47:28.433090: Current learning rate: 0.00897 
+2024-09-05 01:51:54.387755: train_loss -0.7042 
+2024-09-05 01:51:54.389522: val_loss -0.7154 
+2024-09-05 01:51:54.390819: Pseudo dice [0.9558, 0.9496, 0.9742, 0.9354, 0.836, 0.9539, 0.9877] 
+2024-09-05 01:51:54.391881: Epoch time: 265.96 s 
+2024-09-05 01:51:54.393192: Yayy! New best EMA pseudo Dice: 0.9386 
+2024-09-05 01:52:01.975235:  
+2024-09-05 01:52:01.976077: Epoch 115 
+2024-09-05 01:52:01.976865: Current learning rate: 0.00896 
+2024-09-05 01:55:59.958044: train_loss -0.6958 
+2024-09-05 01:55:59.972429: val_loss -0.7037 
+2024-09-05 01:55:59.973250: Pseudo dice [0.9544, 0.9474, 0.9722, 0.9368, 0.8571, 0.9473, 0.9871] 
+2024-09-05 01:55:59.973851: Epoch time: 237.99 s 
+2024-09-05 01:55:59.974458: Yayy! New best EMA pseudo Dice: 0.9391 
+2024-09-05 01:56:08.007931:  
+2024-09-05 01:56:08.008895: Epoch 116 
+2024-09-05 01:56:08.009548: Current learning rate: 0.00895 
+2024-09-05 02:00:27.077677: train_loss -0.7018 
+2024-09-05 02:00:27.089087: val_loss -0.6956 
+2024-09-05 02:00:27.090033: Pseudo dice [0.9557, 0.9486, 0.9696, 0.938, 0.8541, 0.9346, 0.9843] 
+2024-09-05 02:00:27.090947: Epoch time: 259.08 s 
+2024-09-05 02:00:27.091893: Yayy! New best EMA pseudo Dice: 0.9392 
+2024-09-05 02:00:35.791486:  
+2024-09-05 02:00:35.792504: Epoch 117 
+2024-09-05 02:00:35.793500: Current learning rate: 0.00894 
+2024-09-05 02:04:29.311742: train_loss -0.716 
+2024-09-05 02:04:29.313657: val_loss -0.7215 
+2024-09-05 02:04:29.314636: Pseudo dice [0.9586, 0.9499, 0.9748, 0.9392, 0.8371, 0.9469, 0.9875] 
+2024-09-05 02:04:29.315603: Epoch time: 233.52 s 
+2024-09-05 02:04:29.316379: Yayy! New best EMA pseudo Dice: 0.9395 
+2024-09-05 02:04:37.957067:  
+2024-09-05 02:04:37.958205: Epoch 118 
+2024-09-05 02:04:37.959017: Current learning rate: 0.00893 
+2024-09-05 02:09:03.036039: train_loss -0.7245 
+2024-09-05 02:09:03.108124: val_loss -0.7171 
+2024-09-05 02:09:03.226568: Pseudo dice [0.958, 0.9507, 0.9722, 0.9409, 0.8409, 0.9498, 0.9873] 
+2024-09-05 02:09:03.257064: Epoch time: 265.08 s 
+2024-09-05 02:09:03.314618: Yayy! New best EMA pseudo Dice: 0.9398 
+2024-09-05 02:09:13.049713:  
+2024-09-05 02:09:13.050801: Epoch 119 
+2024-09-05 02:09:13.051647: Current learning rate: 0.00892 
+2024-09-05 02:13:30.232911: train_loss -0.6972 
+2024-09-05 02:13:30.235705: val_loss -0.7396 
+2024-09-05 02:13:30.237212: Pseudo dice [0.9535, 0.9434, 0.9652, 0.9389, 0.8518, 0.9475, 0.9886] 
+2024-09-05 02:13:30.239057: Epoch time: 257.19 s 
+2024-09-05 02:13:30.240508: Yayy! New best EMA pseudo Dice: 0.94 
+2024-09-05 02:13:39.567597:  
+2024-09-05 02:13:39.568432: Epoch 120 
+2024-09-05 02:13:39.569061: Current learning rate: 0.00891 
+2024-09-05 02:18:10.721829: train_loss -0.7041 
+2024-09-05 02:18:10.726842: val_loss -0.7264 
+2024-09-05 02:18:10.727715: Pseudo dice [0.9553, 0.9479, 0.9732, 0.9346, 0.8444, 0.9603, 0.9902] 
+2024-09-05 02:18:10.728447: Epoch time: 271.16 s 
+2024-09-05 02:18:10.729846: Yayy! New best EMA pseudo Dice: 0.9404 
+2024-09-05 02:18:20.931734:  
+2024-09-05 02:18:21.000622: Epoch 121 
+2024-09-05 02:18:21.029672: Current learning rate: 0.0089 
+2024-09-05 02:22:32.204762: train_loss -0.7168 
+2024-09-05 02:22:32.206239: val_loss -0.7351 
+2024-09-05 02:22:32.207146: Pseudo dice [0.9601, 0.9514, 0.9766, 0.9442, 0.8333, 0.9484, 0.9863] 
+2024-09-05 02:22:32.207964: Epoch time: 251.28 s 
+2024-09-05 02:22:32.208723: Yayy! New best EMA pseudo Dice: 0.9406 
+2024-09-05 02:22:41.606737:  
+2024-09-05 02:22:41.607771: Epoch 122 
+2024-09-05 02:22:41.608648: Current learning rate: 0.00889 
+2024-09-05 02:27:00.104816: train_loss -0.722 
+2024-09-05 02:27:00.107837: val_loss -0.6979 
+2024-09-05 02:27:00.108615: Pseudo dice [0.9573, 0.9515, 0.9735, 0.9335, 0.8274, 0.9419, 0.9828] 
+2024-09-05 02:27:00.109341: Epoch time: 258.5 s 
+2024-09-05 02:27:02.398405:  
+2024-09-05 02:27:02.401157: Epoch 123 
+2024-09-05 02:27:02.403251: Current learning rate: 0.00889 
+2024-09-05 02:31:41.027454: train_loss -0.7247 
+2024-09-05 02:31:41.112145: val_loss -0.6807 
+2024-09-05 02:31:41.117227: Pseudo dice [0.9583, 0.9469, 0.9738, 0.9306, 0.8367, 0.9545, 0.9858] 
+2024-09-05 02:31:41.117840: Epoch time: 278.63 s 
+2024-09-05 02:31:43.199028:  
+2024-09-05 02:31:43.199929: Epoch 124 
+2024-09-05 02:31:43.200824: Current learning rate: 0.00888 
+2024-09-05 02:35:55.484602: train_loss -0.7027 
+2024-09-05 02:35:55.486887: val_loss -0.7155 
+2024-09-05 02:35:55.487596: Pseudo dice [0.959, 0.952, 0.9771, 0.941, 0.8602, 0.9533, 0.9899] 
+2024-09-05 02:35:55.488257: Epoch time: 252.29 s 
+2024-09-05 02:35:55.488863: Yayy! New best EMA pseudo Dice: 0.9411 
+2024-09-05 02:36:03.899374:  
+2024-09-05 02:36:03.900406: Epoch 125 
+2024-09-05 02:36:03.901078: Current learning rate: 0.00887 
+2024-09-05 02:40:15.822854: train_loss -0.6949 
+2024-09-05 02:40:15.825258: val_loss -0.6865 
+2024-09-05 02:40:15.825916: Pseudo dice [0.9568, 0.9481, 0.9732, 0.9358, 0.8416, 0.9439, 0.9851] 
+2024-09-05 02:40:15.826617: Epoch time: 251.93 s 
+2024-09-05 02:40:18.050791:  
+2024-09-05 02:40:18.051872: Epoch 126 
+2024-09-05 02:40:18.052616: Current learning rate: 0.00886 
+2024-09-05 02:44:51.136711: train_loss -0.7203 
+2024-09-05 02:44:51.139336: val_loss -0.7472 
+2024-09-05 02:44:51.140313: Pseudo dice [0.956, 0.9445, 0.9719, 0.9301, 0.8353, 0.9326, 0.988] 
+2024-09-05 02:44:51.141148: Epoch time: 273.09 s 
+2024-09-05 02:44:53.649613:  
+2024-09-05 02:44:53.651678: Epoch 127 
+2024-09-05 02:44:53.662577: Current learning rate: 0.00885 
+2024-09-05 02:48:58.411976: train_loss -0.7169 
+2024-09-05 02:48:58.414576: val_loss -0.6972 
+2024-09-05 02:48:58.415481: Pseudo dice [0.9587, 0.9497, 0.9765, 0.9424, 0.8141, 0.9459, 0.9839] 
+2024-09-05 02:48:58.416419: Epoch time: 244.77 s 
+2024-09-05 02:49:00.891543:  
+2024-09-05 02:49:00.892745: Epoch 128 
+2024-09-05 02:49:00.893573: Current learning rate: 0.00884 
+2024-09-05 02:53:24.714107: train_loss -0.7094 
+2024-09-05 02:53:24.715424: val_loss -0.7298 
+2024-09-05 02:53:24.716175: Pseudo dice [0.9557, 0.9485, 0.9675, 0.9357, 0.8479, 0.9473, 0.9885] 
+2024-09-05 02:53:24.716907: Epoch time: 263.83 s 
+2024-09-05 02:53:26.983090:  
+2024-09-05 02:53:26.984671: Epoch 129 
+2024-09-05 02:53:26.986069: Current learning rate: 0.00883 
+2024-09-05 02:57:34.001836: train_loss -0.6911 
+2024-09-05 02:57:34.004962: val_loss -0.7401 
+2024-09-05 02:57:34.006409: Pseudo dice [0.9582, 0.9494, 0.9746, 0.9356, 0.8514, 0.9556, 0.9882] 
+2024-09-05 02:57:34.007790: Epoch time: 247.02 s 
+2024-09-05 02:57:36.561189:  
+2024-09-05 02:57:36.574717: Epoch 130 
+2024-09-05 02:57:36.575514: Current learning rate: 0.00882 
+2024-09-05 03:02:00.002078: train_loss -0.6985 
+2024-09-05 03:02:00.003559: val_loss -0.7291 
+2024-09-05 03:02:00.004360: Pseudo dice [0.955, 0.9502, 0.9688, 0.9395, 0.8508, 0.9589, 0.9879] 
+2024-09-05 03:02:00.005077: Epoch time: 263.45 s 
+2024-09-05 03:02:00.005728: Yayy! New best EMA pseudo Dice: 0.9413 
+2024-09-05 03:02:08.444097:  
+2024-09-05 03:02:08.445105: Epoch 131 
+2024-09-05 03:02:08.445818: Current learning rate: 0.00881 
+2024-09-05 03:06:28.411340: train_loss -0.6953 
+2024-09-05 03:06:28.412686: val_loss -0.6922 
+2024-09-05 03:06:28.413476: Pseudo dice [0.9556, 0.95, 0.9712, 0.9279, 0.8526, 0.946, 0.9847] 
+2024-09-05 03:06:28.414314: Epoch time: 259.97 s 
+2024-09-05 03:06:30.692343:  
+2024-09-05 03:06:30.693364: Epoch 132 
+2024-09-05 03:06:30.694971: Current learning rate: 0.0088 
+2024-09-05 03:10:45.009533: train_loss -0.7017 
+2024-09-05 03:10:45.015788: val_loss -0.6868 
+2024-09-05 03:10:45.017494: Pseudo dice [0.9557, 0.952, 0.9715, 0.9368, 0.8187, 0.948, 0.9853] 
+2024-09-05 03:10:45.020862: Epoch time: 254.32 s 
+2024-09-05 03:10:47.387719:  
+2024-09-05 03:10:47.388550: Epoch 133 
+2024-09-05 03:10:47.389199: Current learning rate: 0.00879 
+2024-09-05 03:14:55.290151: train_loss -0.7178 
+2024-09-05 03:14:55.295564: val_loss -0.7298 
+2024-09-05 03:14:55.296592: Pseudo dice [0.9576, 0.9481, 0.974, 0.9368, 0.8359, 0.9596, 0.9844] 
+2024-09-05 03:14:55.299008: Epoch time: 247.91 s 
+2024-09-05 03:14:57.451571:  
+2024-09-05 03:14:57.453404: Epoch 134 
+2024-09-05 03:14:57.454780: Current learning rate: 0.00879 
+2024-09-05 03:19:14.816658: train_loss -0.706 
+2024-09-05 03:19:14.833705: val_loss -0.7174 
+2024-09-05 03:19:14.834623: Pseudo dice [0.96, 0.9512, 0.9735, 0.9299, 0.8533, 0.9336, 0.9845] 
+2024-09-05 03:19:14.835804: Epoch time: 257.37 s 
+2024-09-05 03:19:18.265651:  
+2024-09-05 03:19:18.266546: Epoch 135 
+2024-09-05 03:19:18.267307: Current learning rate: 0.00878 
+2024-09-05 03:23:24.036807: train_loss -0.7158 
+2024-09-05 03:23:24.049974: val_loss -0.7058 
+2024-09-05 03:23:24.051357: Pseudo dice [0.9582, 0.9509, 0.9745, 0.937, 0.8274, 0.9526, 0.9851] 
+2024-09-05 03:23:24.052333: Epoch time: 245.78 s 
+2024-09-05 03:23:26.407350:  
+2024-09-05 03:23:26.408226: Epoch 136 
+2024-09-05 03:23:26.408931: Current learning rate: 0.00877 
+2024-09-05 03:27:25.095440: train_loss -0.7021 
+2024-09-05 03:27:25.108187: val_loss -0.6913 
+2024-09-05 03:27:25.109202: Pseudo dice [0.9543, 0.9503, 0.9698, 0.9369, 0.8428, 0.9574, 0.9901] 
+2024-09-05 03:27:25.109944: Epoch time: 238.69 s 
+2024-09-05 03:27:27.760906:  
+2024-09-05 03:27:27.761806: Epoch 137 
+2024-09-05 03:27:27.762467: Current learning rate: 0.00876 
+2024-09-05 03:32:10.320718: train_loss -0.7064 
+2024-09-05 03:32:10.336483: val_loss -0.7063 
+2024-09-05 03:32:10.337295: Pseudo dice [0.96, 0.9543, 0.9757, 0.9394, 0.8387, 0.9589, 0.987] 
+2024-09-05 03:32:10.338065: Epoch time: 282.56 s 
+2024-09-05 03:32:10.338785: Yayy! New best EMA pseudo Dice: 0.9417 
+2024-09-05 03:32:18.678455:  
+2024-09-05 03:32:18.679561: Epoch 138 
+2024-09-05 03:32:18.680495: Current learning rate: 0.00875 
+2024-09-05 03:36:32.611626: train_loss -0.7278 
+2024-09-05 03:36:32.627017: val_loss -0.7545 
+2024-09-05 03:36:32.627944: Pseudo dice [0.9613, 0.9522, 0.9769, 0.9405, 0.8612, 0.9638, 0.9906] 
+2024-09-05 03:36:32.628702: Epoch time: 253.94 s 
+2024-09-05 03:36:32.629360: Yayy! New best EMA pseudo Dice: 0.9424 
+2024-09-05 03:36:41.104943:  
+2024-09-05 03:36:41.106069: Epoch 139 
+2024-09-05 03:36:41.106797: Current learning rate: 0.00874 
+2024-09-05 03:41:07.151157: train_loss -0.7325 
+2024-09-05 03:41:07.155334: val_loss -0.782 
+2024-09-05 03:41:07.156312: Pseudo dice [0.9621, 0.9534, 0.9792, 0.9381, 0.8639, 0.9542, 0.9894] 
+2024-09-05 03:41:07.157152: Epoch time: 266.05 s 
+2024-09-05 03:41:07.157667: Yayy! New best EMA pseudo Dice: 0.9431 
+2024-09-05 03:41:15.896826:  
+2024-09-05 03:41:15.897581: Epoch 140 
+2024-09-05 03:41:15.898150: Current learning rate: 0.00873 
+2024-09-05 03:45:27.747180: train_loss -0.7305 
+2024-09-05 03:45:27.749628: val_loss -0.7013 
+2024-09-05 03:45:27.750582: Pseudo dice [0.9604, 0.9527, 0.9758, 0.9415, 0.8422, 0.9533, 0.9881] 
+2024-09-05 03:45:27.751281: Epoch time: 251.85 s 
+2024-09-05 03:45:27.751946: Yayy! New best EMA pseudo Dice: 0.9432 
+2024-09-05 03:45:37.103988:  
+2024-09-05 03:45:37.105145: Epoch 141 
+2024-09-05 03:45:37.106460: Current learning rate: 0.00872 
+2024-09-05 03:49:39.158767: train_loss -0.725 
+2024-09-05 03:49:39.174775: val_loss -0.7194 
+2024-09-05 03:49:39.175910: Pseudo dice [0.9608, 0.9502, 0.9772, 0.9404, 0.8572, 0.9485, 0.9877] 
+2024-09-05 03:49:39.176831: Epoch time: 242.06 s 
+2024-09-05 03:49:39.177632: Yayy! New best EMA pseudo Dice: 0.9435 
+2024-09-05 03:49:47.427225:  
+2024-09-05 03:49:47.428182: Epoch 142 
+2024-09-05 03:49:47.428828: Current learning rate: 0.00871 
+2024-09-05 03:54:05.059782: train_loss -0.6999 
+2024-09-05 03:54:05.061288: val_loss -0.6901 
+2024-09-05 03:54:05.062765: Pseudo dice [0.9579, 0.9468, 0.9711, 0.9374, 0.8121, 0.9602, 0.9864] 
+2024-09-05 03:54:05.063699: Epoch time: 257.64 s 
+2024-09-05 03:54:07.432936:  
+2024-09-05 03:54:07.447243: Epoch 143 
+2024-09-05 03:54:07.449144: Current learning rate: 0.0087 
+2024-09-05 03:58:02.718345: train_loss -0.7167 
+2024-09-05 03:58:02.732136: val_loss -0.7259 
+2024-09-05 03:58:02.733265: Pseudo dice [0.9613, 0.9501, 0.9757, 0.9383, 0.8494, 0.9545, 0.9882] 
+2024-09-05 03:58:02.734348: Epoch time: 235.29 s 
+2024-09-05 03:58:05.184630:  
+2024-09-05 03:58:05.185528: Epoch 144 
+2024-09-05 03:58:05.186122: Current learning rate: 0.00869 
+2024-09-05 04:02:24.104208: train_loss -0.7207 
+2024-09-05 04:02:24.106761: val_loss -0.6826 
+2024-09-05 04:02:24.107471: Pseudo dice [0.9573, 0.9526, 0.9718, 0.9365, 0.8578, 0.9492, 0.9864] 
+2024-09-05 04:02:24.108085: Epoch time: 258.92 s 
+2024-09-05 04:02:26.440330:  
+2024-09-05 04:02:26.441585: Epoch 145 
+2024-09-05 04:02:26.442538: Current learning rate: 0.00868 
+2024-09-05 04:06:54.380394: train_loss -0.7281 
+2024-09-05 04:06:54.384151: val_loss -0.7149 
+2024-09-05 04:06:54.385452: Pseudo dice [0.9574, 0.9525, 0.9724, 0.9359, 0.8248, 0.9598, 0.9857] 
+2024-09-05 04:06:54.386642: Epoch time: 267.95 s 
+2024-09-05 04:06:57.227664:  
+2024-09-05 04:06:57.228609: Epoch 146 
+2024-09-05 04:06:57.229265: Current learning rate: 0.00868 
+2024-09-05 04:11:14.062163: train_loss -0.7334 
+2024-09-05 04:11:14.080070: val_loss -0.7348 
+2024-09-05 04:11:14.080923: Pseudo dice [0.9583, 0.9545, 0.9747, 0.9432, 0.8573, 0.9622, 0.9897] 
+2024-09-05 04:11:14.081839: Epoch time: 256.84 s 
+2024-09-05 04:11:14.082604: Yayy! New best EMA pseudo Dice: 0.9437 
+2024-09-05 04:11:23.240465:  
+2024-09-05 04:11:23.241542: Epoch 147 
+2024-09-05 04:11:23.242357: Current learning rate: 0.00867 
+2024-09-05 04:15:41.032583: train_loss -0.7134 
+2024-09-05 04:15:41.047054: val_loss -0.7268 
+2024-09-05 04:15:41.047879: Pseudo dice [0.9576, 0.9508, 0.9739, 0.9428, 0.8532, 0.9605, 0.9889] 
+2024-09-05 04:15:41.048610: Epoch time: 257.8 s 
+2024-09-05 04:15:41.049212: Yayy! New best EMA pseudo Dice: 0.944 
+2024-09-05 04:15:50.222366:  
+2024-09-05 04:15:50.224131: Epoch 148 
+2024-09-05 04:15:50.224787: Current learning rate: 0.00866 
+2024-09-05 04:20:14.731270: train_loss -0.7278 
+2024-09-05 04:20:14.748465: val_loss -0.7199 
+2024-09-05 04:20:14.749603: Pseudo dice [0.9587, 0.9534, 0.9767, 0.9427, 0.839, 0.9622, 0.9869] 
+2024-09-05 04:20:14.750749: Epoch time: 264.51 s 
+2024-09-05 04:20:14.751683: Yayy! New best EMA pseudo Dice: 0.9442 
+2024-09-05 04:20:24.436531:  
+2024-09-05 04:20:24.437541: Epoch 149 
+2024-09-05 04:20:24.438293: Current learning rate: 0.00865 
+2024-09-05 04:24:50.775753: train_loss -0.7076 
+2024-09-05 04:24:50.799974: val_loss -0.7113 
+2024-09-05 04:24:50.800963: Pseudo dice [0.9567, 0.9522, 0.9699, 0.9424, 0.8591, 0.9579, 0.9879] 
+2024-09-05 04:24:50.802020: Epoch time: 266.34 s 
+2024-09-05 04:24:57.550898: Yayy! New best EMA pseudo Dice: 0.9444 
+2024-09-05 04:25:05.215537:  
+2024-09-05 04:25:05.216326: Epoch 150 
+2024-09-05 04:25:05.216887: Current learning rate: 0.00864 
+2024-09-05 04:28:50.565195: train_loss -0.7242 
+2024-09-05 04:28:50.578622: val_loss -0.7318 
+2024-09-05 04:28:50.580344: Pseudo dice [0.9567, 0.9522, 0.9723, 0.943, 0.8438, 0.9596, 0.9879] 
+2024-09-05 04:28:50.581493: Epoch time: 225.35 s 
+2024-09-05 04:28:50.582593: Yayy! New best EMA pseudo Dice: 0.9445 
+2024-09-05 04:29:03.088023:  
+2024-09-05 04:29:03.089172: Epoch 151 
+2024-09-05 04:29:03.090060: Current learning rate: 0.00863 
+2024-09-05 04:33:18.036907: train_loss -0.7106 
+2024-09-05 04:33:18.043236: val_loss -0.7512 
+2024-09-05 04:33:18.044117: Pseudo dice [0.9551, 0.9497, 0.9695, 0.9362, 0.8424, 0.9545, 0.989] 
+2024-09-05 04:33:18.044883: Epoch time: 254.95 s 
+2024-09-05 04:33:20.322969:  
+2024-09-05 04:33:20.324186: Epoch 152 
+2024-09-05 04:33:20.324903: Current learning rate: 0.00862 
+2024-09-05 04:37:46.187275: train_loss -0.7133 
+2024-09-05 04:37:46.199042: val_loss -0.7333 
+2024-09-05 04:37:46.200229: Pseudo dice [0.9578, 0.9511, 0.9725, 0.9389, 0.8581, 0.9533, 0.9874] 
+2024-09-05 04:37:46.201067: Epoch time: 265.87 s 
+2024-09-05 04:37:48.561749:  
+2024-09-05 04:37:48.563017: Epoch 153 
+2024-09-05 04:37:48.564327: Current learning rate: 0.00861 
+2024-09-05 04:42:21.507703: train_loss -0.7246 
+2024-09-05 04:42:21.520274: val_loss -0.7495 
+2024-09-05 04:42:21.522043: Pseudo dice [0.9592, 0.951, 0.9748, 0.9396, 0.8553, 0.955, 0.9883] 
+2024-09-05 04:42:21.523290: Epoch time: 272.95 s 
+2024-09-05 04:42:21.524337: Yayy! New best EMA pseudo Dice: 0.9446 
+2024-09-05 04:42:30.876923:  
+2024-09-05 04:42:30.878459: Epoch 154 
+2024-09-05 04:42:30.880438: Current learning rate: 0.0086 
+2024-09-05 04:46:39.556049: train_loss -0.7117 
+2024-09-05 04:46:39.570054: val_loss -0.7161 
+2024-09-05 04:46:39.571044: Pseudo dice [0.9543, 0.9491, 0.9703, 0.9363, 0.8253, 0.9573, 0.9858] 
+2024-09-05 04:46:39.571815: Epoch time: 248.68 s 
+2024-09-05 04:46:42.042052:  
+2024-09-05 04:46:42.043226: Epoch 155 
+2024-09-05 04:46:42.044189: Current learning rate: 0.00859 
+2024-09-05 04:50:56.870792: train_loss -0.71 
+2024-09-05 04:50:56.872856: val_loss -0.7211 
+2024-09-05 04:50:56.873434: Pseudo dice [0.958, 0.9501, 0.9715, 0.9404, 0.8436, 0.9512, 0.9869] 
+2024-09-05 04:50:56.874071: Epoch time: 254.83 s 
+2024-09-05 04:50:59.021909:  
+2024-09-05 04:50:59.034598: Epoch 156 
+2024-09-05 04:50:59.035733: Current learning rate: 0.00858 
+2024-09-05 04:55:05.235263: train_loss -0.6928 
+2024-09-05 04:55:05.240200: val_loss -0.7559 
+2024-09-05 04:55:05.241217: Pseudo dice [0.9576, 0.9524, 0.9706, 0.9386, 0.8313, 0.9596, 0.985] 
+2024-09-05 04:55:05.242187: Epoch time: 246.22 s 
+2024-09-05 04:55:08.854886:  
+2024-09-05 04:55:08.856678: Epoch 157 
+2024-09-05 04:55:08.858013: Current learning rate: 0.00858 
+2024-09-05 04:59:15.720236: train_loss -0.7087 
+2024-09-05 04:59:15.734694: val_loss -0.6603 
+2024-09-05 04:59:15.735381: Pseudo dice [0.9588, 0.9493, 0.9733, 0.9428, 0.8411, 0.9564, 0.9873] 
+2024-09-05 04:59:15.736203: Epoch time: 246.87 s 
+2024-09-05 04:59:18.049399:  
+2024-09-05 04:59:18.051012: Epoch 158 
+2024-09-05 04:59:18.051631: Current learning rate: 0.00857 
+2024-09-05 05:03:47.406560: train_loss -0.6992 
+2024-09-05 05:03:47.408013: val_loss -0.6565 
+2024-09-05 05:03:47.408938: Pseudo dice [0.958, 0.9518, 0.9705, 0.9331, 0.82, 0.9133, 0.9791] 
+2024-09-05 05:03:47.410219: Epoch time: 269.36 s 
+2024-09-05 05:03:49.664395:  
+2024-09-05 05:03:49.665412: Epoch 159 
+2024-09-05 05:03:49.665994: Current learning rate: 0.00856 
+2024-09-05 05:08:16.944342: train_loss -0.7017 
+2024-09-05 05:08:16.958479: val_loss -0.6929 
+2024-09-05 05:08:16.959359: Pseudo dice [0.9537, 0.9481, 0.9728, 0.9354, 0.8233, 0.9422, 0.9822] 
+2024-09-05 05:08:16.960319: Epoch time: 267.28 s 
+2024-09-05 05:08:19.527735:  
+2024-09-05 05:08:19.528698: Epoch 160 
+2024-09-05 05:08:19.529347: Current learning rate: 0.00855 
+2024-09-05 05:12:54.067979: train_loss -0.688 
+2024-09-05 05:12:54.084380: val_loss -0.705 
+2024-09-05 05:12:54.085477: Pseudo dice [0.9529, 0.9464, 0.9696, 0.9327, 0.8239, 0.9392, 0.9843] 
+2024-09-05 05:12:54.086504: Epoch time: 274.56 s 
+2024-09-05 05:12:56.794091:  
+2024-09-05 05:12:56.795063: Epoch 161 
+2024-09-05 05:12:56.795822: Current learning rate: 0.00854 
+2024-09-05 05:17:15.172951: train_loss -0.7192 
+2024-09-05 05:17:15.202838: val_loss -0.6932 
+2024-09-05 05:17:15.203670: Pseudo dice [0.9607, 0.9515, 0.9746, 0.9347, 0.8514, 0.9527, 0.9867] 
+2024-09-05 05:17:15.204590: Epoch time: 258.38 s 
+2024-09-05 05:17:17.425633:  
+2024-09-05 05:17:17.426882: Epoch 162 
+2024-09-05 05:17:17.427634: Current learning rate: 0.00853 
+2024-09-05 05:21:37.629490: train_loss -0.7198 
+2024-09-05 05:21:37.650313: val_loss -0.7347 
+2024-09-05 05:21:37.658698: Pseudo dice [0.9609, 0.9535, 0.9778, 0.944, 0.8613, 0.963, 0.9891] 
+2024-09-05 05:21:37.659543: Epoch time: 260.21 s 
+2024-09-05 05:21:40.151760:  
+2024-09-05 05:21:40.160730: Epoch 163 
+2024-09-05 05:21:40.161724: Current learning rate: 0.00852 
+2024-09-05 05:25:55.160012: train_loss -0.7178 
+2024-09-05 05:25:55.163786: val_loss -0.7235 
+2024-09-05 05:25:55.165128: Pseudo dice [0.9592, 0.9525, 0.9761, 0.9404, 0.8597, 0.9537, 0.9865] 
+2024-09-05 05:25:55.167833: Epoch time: 255.01 s 
+2024-09-05 05:25:57.971824:  
+2024-09-05 05:25:57.973070: Epoch 164 
+2024-09-05 05:25:57.973912: Current learning rate: 0.00851 
+2024-09-05 05:30:34.463081: train_loss -0.7101 
+2024-09-05 05:30:34.468208: val_loss -0.716 
+2024-09-05 05:30:34.469114: Pseudo dice [0.9568, 0.9524, 0.9724, 0.9396, 0.8231, 0.9595, 0.9868] 
+2024-09-05 05:30:34.469968: Epoch time: 276.5 s 
+2024-09-05 05:30:37.340467:  
+2024-09-05 05:30:37.341633: Epoch 165 
+2024-09-05 05:30:37.342873: Current learning rate: 0.0085 
+2024-09-05 05:34:46.620669: train_loss -0.7265 
+2024-09-05 05:34:46.640160: val_loss -0.7185 
+2024-09-05 05:34:46.641001: Pseudo dice [0.961, 0.9558, 0.9773, 0.9431, 0.8589, 0.9641, 0.9908] 
+2024-09-05 05:34:46.641837: Epoch time: 249.28 s 
+2024-09-05 05:34:49.054760:  
+2024-09-05 05:34:49.055801: Epoch 166 
+2024-09-05 05:34:49.056596: Current learning rate: 0.00849 
+2024-09-05 05:39:01.984835: train_loss -0.7343 
+2024-09-05 05:39:02.000361: val_loss -0.7577 
+2024-09-05 05:39:02.001442: Pseudo dice [0.9598, 0.9519, 0.9761, 0.9425, 0.8454, 0.9613, 0.9884] 
+2024-09-05 05:39:02.002495: Epoch time: 252.94 s 
+2024-09-05 05:39:04.326895:  
+2024-09-05 05:39:04.328124: Epoch 167 
+2024-09-05 05:39:04.328988: Current learning rate: 0.00848 
+2024-09-05 05:43:33.559618: train_loss -0.732 
+2024-09-05 05:43:33.575932: val_loss -0.746 
+2024-09-05 05:43:33.576680: Pseudo dice [0.96, 0.9535, 0.9758, 0.9438, 0.84, 0.9639, 0.9894] 
+2024-09-05 05:43:33.577701: Epoch time: 269.24 s 
+2024-09-05 05:43:37.154890:  
+2024-09-05 05:43:37.155849: Epoch 168 
+2024-09-05 05:43:37.156608: Current learning rate: 0.00847 
+2024-09-05 05:47:46.750987: train_loss -0.7257 
+2024-09-05 05:47:46.753395: val_loss -0.7323 
+2024-09-05 05:47:46.754058: Pseudo dice [0.9618, 0.9523, 0.9772, 0.9436, 0.8468, 0.9633, 0.9895] 
+2024-09-05 05:47:46.754926: Epoch time: 249.6 s 
+2024-09-05 05:47:49.220923:  
+2024-09-05 05:47:49.222101: Epoch 169 
+2024-09-05 05:47:49.222984: Current learning rate: 0.00847 
+2024-09-05 05:52:17.530054: train_loss -0.7102 
+2024-09-05 05:52:17.532665: val_loss -0.7087 
+2024-09-05 05:52:17.533477: Pseudo dice [0.9604, 0.9541, 0.975, 0.9412, 0.8548, 0.9635, 0.9886] 
+2024-09-05 05:52:17.534205: Epoch time: 268.31 s 
+2024-09-05 05:52:17.534851: Yayy! New best EMA pseudo Dice: 0.9449 
+2024-09-05 05:52:26.731272:  
+2024-09-05 05:52:26.732196: Epoch 170 
+2024-09-05 05:52:26.732852: Current learning rate: 0.00846 
+2024-09-05 05:56:35.230042: train_loss -0.7292 
+2024-09-05 05:56:35.235672: val_loss -0.6697 
+2024-09-05 05:56:35.236619: Pseudo dice [0.9577, 0.954, 0.972, 0.9382, 0.8127, 0.9604, 0.9867] 
+2024-09-05 05:56:35.237467: Epoch time: 248.5 s 
+2024-09-05 05:56:37.869203:  
+2024-09-05 05:56:37.870240: Epoch 171 
+2024-09-05 05:56:37.871169: Current learning rate: 0.00845 
+2024-09-05 06:01:00.291316: train_loss -0.7159 
+2024-09-05 06:01:00.293823: val_loss -0.7144 
+2024-09-05 06:01:00.294826: Pseudo dice [0.9619, 0.9555, 0.9778, 0.9442, 0.8396, 0.9528, 0.9862] 
+2024-09-05 06:01:00.295568: Epoch time: 262.43 s 
+2024-09-05 06:01:02.502797:  
+2024-09-05 06:01:02.509922: Epoch 172 
+2024-09-05 06:01:02.510809: Current learning rate: 0.00844 
+2024-09-05 06:05:35.743873: train_loss -0.7307 
+2024-09-05 06:05:35.758671: val_loss -0.7541 
+2024-09-05 06:05:35.759543: Pseudo dice [0.9599, 0.9525, 0.9748, 0.9404, 0.8199, 0.9517, 0.9845] 
+2024-09-05 06:05:35.760252: Epoch time: 273.25 s 
+2024-09-05 06:05:38.258042:  
+2024-09-05 06:05:38.259026: Epoch 173 
+2024-09-05 06:05:38.259851: Current learning rate: 0.00843 
+2024-09-05 06:09:47.529916: train_loss -0.73 
+2024-09-05 06:09:47.552312: val_loss -0.7436 
+2024-09-05 06:09:47.553282: Pseudo dice [0.9629, 0.955, 0.9773, 0.9442, 0.863, 0.9655, 0.9901] 
+2024-09-05 06:09:47.554279: Epoch time: 249.28 s 
+2024-09-05 06:09:49.666810:  
+2024-09-05 06:09:49.667950: Epoch 174 
+2024-09-05 06:09:49.679840: Current learning rate: 0.00842 
+2024-09-05 06:13:55.562871: train_loss -0.7139 
+2024-09-05 06:13:55.564468: val_loss -0.6909 
+2024-09-05 06:13:55.565547: Pseudo dice [0.959, 0.9501, 0.9757, 0.9392, 0.8492, 0.9539, 0.9889] 
+2024-09-05 06:13:55.566752: Epoch time: 245.9 s 
+2024-09-05 06:13:57.775856:  
+2024-09-05 06:13:57.776915: Epoch 175 
+2024-09-05 06:13:57.777527: Current learning rate: 0.00841 
+2024-09-05 06:18:09.458808: train_loss -0.7212 
+2024-09-05 06:18:09.472120: val_loss -0.7079 
+2024-09-05 06:18:09.472957: Pseudo dice [0.9578, 0.9504, 0.9714, 0.9338, 0.8233, 0.9602, 0.9865] 
+2024-09-05 06:18:09.473735: Epoch time: 251.69 s 
+2024-09-05 06:18:11.960035:  
+2024-09-05 06:18:11.961174: Epoch 176 
+2024-09-05 06:18:11.962186: Current learning rate: 0.0084 
+2024-09-05 06:22:35.338854: train_loss -0.7312 
+2024-09-05 06:22:35.341667: val_loss -0.7036 
+2024-09-05 06:22:35.342402: Pseudo dice [0.9564, 0.9537, 0.9701, 0.9459, 0.8621, 0.9519, 0.9894] 
+2024-09-05 06:22:35.343147: Epoch time: 263.38 s 
+2024-09-05 06:22:37.811287:  
+2024-09-05 06:22:37.816554: Epoch 177 
+2024-09-05 06:22:37.817784: Current learning rate: 0.00839 
+2024-09-05 06:27:04.597347: train_loss -0.7089 
+2024-09-05 06:27:04.599768: val_loss -0.7157 
+2024-09-05 06:27:04.600525: Pseudo dice [0.9611, 0.9558, 0.9767, 0.9435, 0.8509, 0.9591, 0.9898] 
+2024-09-05 06:27:04.601187: Epoch time: 266.79 s 
+2024-09-05 06:27:04.601801: Yayy! New best EMA pseudo Dice: 0.945 
+2024-09-05 06:27:12.945230:  
+2024-09-05 06:27:12.946094: Epoch 178 
+2024-09-05 06:27:12.946937: Current learning rate: 0.00838 
+2024-09-05 06:31:31.763850: train_loss -0.7007 
+2024-09-05 06:31:31.766374: val_loss -0.7155 
+2024-09-05 06:31:31.767488: Pseudo dice [0.9574, 0.9457, 0.9724, 0.9319, 0.8609, 0.9588, 0.9892] 
+2024-09-05 06:31:31.768506: Epoch time: 258.82 s 
+2024-09-05 06:31:31.769296: Yayy! New best EMA pseudo Dice: 0.945 
+2024-09-05 06:31:42.700526:  
+2024-09-05 06:31:42.701968: Epoch 179 
+2024-09-05 06:31:42.702995: Current learning rate: 0.00837 
+2024-09-05 06:35:51.507354: train_loss -0.7234 
+2024-09-05 06:35:51.510065: val_loss -0.6943 
+2024-09-05 06:35:51.511158: Pseudo dice [0.9587, 0.9534, 0.9732, 0.9358, 0.824, 0.9535, 0.9849] 
+2024-09-05 06:35:51.512237: Epoch time: 248.81 s 
+2024-09-05 06:35:53.883847:  
+2024-09-05 06:35:53.884940: Epoch 180 
+2024-09-05 06:35:53.885740: Current learning rate: 0.00836 
+2024-09-05 06:40:10.903544: train_loss -0.7256 
+2024-09-05 06:40:10.915359: val_loss -0.7944 
+2024-09-05 06:40:10.916442: Pseudo dice [0.9599, 0.9501, 0.9765, 0.938, 0.8677, 0.9622, 0.9905] 
+2024-09-05 06:40:10.917725: Epoch time: 257.02 s 
+2024-09-05 06:40:10.918648: Yayy! New best EMA pseudo Dice: 0.9451 
+2024-09-05 06:40:22.060162:  
+2024-09-05 06:40:22.061146: Epoch 181 
+2024-09-05 06:40:22.061737: Current learning rate: 0.00836 
+2024-09-05 06:44:32.292855: train_loss -0.7023 
+2024-09-05 06:44:32.315485: val_loss -0.727 
+2024-09-05 06:44:32.316855: Pseudo dice [0.9521, 0.9448, 0.9652, 0.9261, 0.8256, 0.9561, 0.9838] 
+2024-09-05 06:44:32.318197: Epoch time: 250.24 s 
+2024-09-05 06:44:35.109599:  
+2024-09-05 06:44:35.110771: Epoch 182 
+2024-09-05 06:44:35.111665: Current learning rate: 0.00835 
+2024-09-05 06:49:00.583512: train_loss -0.6874 
+2024-09-05 06:49:00.601601: val_loss -0.703 
+2024-09-05 06:49:00.602538: Pseudo dice [0.9531, 0.9449, 0.9703, 0.9265, 0.8147, 0.9586, 0.9889] 
+2024-09-05 06:49:00.603536: Epoch time: 265.48 s 
+2024-09-05 06:49:03.230735:  
+2024-09-05 06:49:03.246572: Epoch 183 
+2024-09-05 06:49:03.247226: Current learning rate: 0.00834 
+2024-09-05 06:53:25.849182: train_loss -0.6798 
+2024-09-05 06:53:25.850672: val_loss -0.6667 
+2024-09-05 06:53:25.851490: Pseudo dice [0.9482, 0.945, 0.9641, 0.9284, 0.7931, 0.952, 0.983] 
+2024-09-05 06:53:25.852210: Epoch time: 262.62 s 
+2024-09-05 06:53:28.316292:  
+2024-09-05 06:53:28.317664: Epoch 184 
+2024-09-05 06:53:28.319738: Current learning rate: 0.00833 
+2024-09-05 06:57:52.102338: train_loss -0.6557 
+2024-09-05 06:57:52.109751: val_loss -0.677 
+2024-09-05 06:57:52.110547: Pseudo dice [0.9478, 0.9424, 0.9665, 0.9269, 0.8063, 0.9428, 0.9808] 
+2024-09-05 06:57:52.111295: Epoch time: 263.79 s 
+2024-09-05 06:57:54.618802:  
+2024-09-05 06:57:54.620246: Epoch 185 
+2024-09-05 06:57:54.621486: Current learning rate: 0.00832 
+2024-09-05 07:02:10.544934: train_loss -0.6718 
+2024-09-05 07:02:10.548110: val_loss -0.6232 
+2024-09-05 07:02:10.549015: Pseudo dice [0.9395, 0.9431, 0.9255, 0.921, 0.8186, 0.9293, 0.9799] 
+2024-09-05 07:02:10.550016: Epoch time: 255.93 s 
+2024-09-05 07:02:12.604955:  
+2024-09-05 07:02:12.605994: Epoch 186 
+2024-09-05 07:02:12.606839: Current learning rate: 0.00831 
+2024-09-05 07:06:33.148760: train_loss -0.6656 
+2024-09-05 07:06:33.151691: val_loss -0.674 
+2024-09-05 07:06:33.166096: Pseudo dice [0.9517, 0.9472, 0.9704, 0.9306, 0.816, 0.9533, 0.9842] 
+2024-09-05 07:06:33.167036: Epoch time: 260.55 s 
+2024-09-05 07:06:35.890461:  
+2024-09-05 07:06:35.891324: Epoch 187 
+2024-09-05 07:06:35.891913: Current learning rate: 0.0083 
+2024-09-05 07:10:53.487675: train_loss -0.694 
+2024-09-05 07:10:53.490724: val_loss -0.6956 
+2024-09-05 07:10:53.491322: Pseudo dice [0.9533, 0.9469, 0.9667, 0.9289, 0.8508, 0.9578, 0.9884] 
+2024-09-05 07:10:53.492017: Epoch time: 257.6 s 
+2024-09-05 07:10:55.597616:  
+2024-09-05 07:10:55.598430: Epoch 188 
+2024-09-05 07:10:55.599140: Current learning rate: 0.00829 
+2024-09-05 07:15:18.040254: train_loss -0.7078 
+2024-09-05 07:15:18.043059: val_loss -0.715 
+2024-09-05 07:15:18.043901: Pseudo dice [0.9561, 0.9506, 0.9709, 0.9409, 0.8089, 0.9577, 0.9862] 
+2024-09-05 07:15:18.044618: Epoch time: 262.45 s 
+2024-09-05 07:15:20.380877:  
+2024-09-05 07:15:20.382144: Epoch 189 
+2024-09-05 07:15:20.383062: Current learning rate: 0.00828 
+2024-09-05 07:19:46.486942: train_loss -0.7034 
+2024-09-05 07:19:46.503037: val_loss -0.7224 
+2024-09-05 07:19:46.504374: Pseudo dice [0.9564, 0.9498, 0.9752, 0.9328, 0.8354, 0.9466, 0.9841] 
+2024-09-05 07:19:46.505669: Epoch time: 266.11 s 
+2024-09-05 07:19:48.606486:  
+2024-09-05 07:19:48.607700: Epoch 190 
+2024-09-05 07:19:48.608498: Current learning rate: 0.00827 
+2024-09-05 07:24:02.785319: train_loss -0.711 
+2024-09-05 07:24:02.810062: val_loss -0.7339 
+2024-09-05 07:24:02.811123: Pseudo dice [0.9551, 0.9489, 0.9731, 0.9347, 0.8569, 0.958, 0.988] 
+2024-09-05 07:24:02.811852: Epoch time: 254.18 s 
+2024-09-05 07:24:05.535685:  
+2024-09-05 07:24:05.536786: Epoch 191 
+2024-09-05 07:24:05.537583: Current learning rate: 0.00826 
+2024-09-05 07:28:37.952503: train_loss -0.7186 
+2024-09-05 07:28:37.962081: val_loss -0.7248 
+2024-09-05 07:28:37.962940: Pseudo dice [0.9581, 0.9517, 0.9759, 0.9384, 0.8539, 0.9594, 0.987] 
+2024-09-05 07:28:37.964132: Epoch time: 272.42 s 
+2024-09-05 07:28:40.163987:  
+2024-09-05 07:28:40.165572: Epoch 192 
+2024-09-05 07:28:40.166826: Current learning rate: 0.00825 
+2024-09-05 07:32:47.566516: train_loss -0.7107 
+2024-09-05 07:32:47.580122: val_loss -0.6774 
+2024-09-05 07:32:47.581930: Pseudo dice [0.9587, 0.9526, 0.9756, 0.9386, 0.8439, 0.9434, 0.9846] 
+2024-09-05 07:32:47.584150: Epoch time: 247.41 s 
+2024-09-05 07:32:50.178530:  
+2024-09-05 07:32:50.179701: Epoch 193 
+2024-09-05 07:32:50.180884: Current learning rate: 0.00824 
+2024-09-05 07:37:22.827636: train_loss -0.7087 
+2024-09-05 07:37:22.836481: val_loss -0.7028 
+2024-09-05 07:37:22.837754: Pseudo dice [0.9577, 0.9505, 0.9728, 0.9398, 0.8369, 0.9565, 0.9869] 
+2024-09-05 07:37:22.838750: Epoch time: 272.66 s 
+2024-09-05 07:37:26.415017:  
+2024-09-05 07:37:26.415964: Epoch 194 
+2024-09-05 07:37:26.416676: Current learning rate: 0.00824 
+2024-09-05 07:41:32.479230: train_loss -0.7162 
+2024-09-05 07:41:32.481534: val_loss -0.7109 
+2024-09-05 07:41:32.482499: Pseudo dice [0.9489, 0.9467, 0.9601, 0.9303, 0.8355, 0.954, 0.9893] 
+2024-09-05 07:41:32.483776: Epoch time: 246.07 s 
+2024-09-05 07:41:35.048364:  
+2024-09-05 07:41:35.049563: Epoch 195 
+2024-09-05 07:41:35.050519: Current learning rate: 0.00823 
+2024-09-05 07:46:07.120910: train_loss -0.6985 
+2024-09-05 07:46:07.137904: val_loss -0.7575 
+2024-09-05 07:46:07.139334: Pseudo dice [0.9553, 0.9479, 0.9711, 0.9373, 0.8394, 0.9616, 0.9875] 
+2024-09-05 07:46:07.140856: Epoch time: 272.08 s 
+2024-09-05 07:46:09.998978:  
+2024-09-05 07:46:09.999873: Epoch 196 
+2024-09-05 07:46:10.000980: Current learning rate: 0.00822 
+2024-09-05 07:50:19.214230: train_loss -0.7172 
+2024-09-05 07:50:19.217213: val_loss -0.6835 
+2024-09-05 07:50:19.218486: Pseudo dice [0.9585, 0.953, 0.9731, 0.9432, 0.8291, 0.9579, 0.9848] 
+2024-09-05 07:50:19.219594: Epoch time: 249.22 s 
+2024-09-05 07:50:21.413366:  
+2024-09-05 07:50:21.414409: Epoch 197 
+2024-09-05 07:50:21.415295: Current learning rate: 0.00821 
+2024-09-05 07:54:51.335896: train_loss -0.7354 
+2024-09-05 07:54:51.338834: val_loss -0.7341 
+2024-09-05 07:54:51.364189: Pseudo dice [0.958, 0.951, 0.9692, 0.9414, 0.8659, 0.9628, 0.9901] 
+2024-09-05 07:54:51.365081: Epoch time: 269.93 s 
+2024-09-05 07:54:55.546695:  
+2024-09-05 07:54:55.547556: Epoch 198 
+2024-09-05 07:54:55.548226: Current learning rate: 0.0082 
+2024-09-05 07:59:00.407340: train_loss -0.7136 
+2024-09-05 07:59:00.409824: val_loss -0.7228 
+2024-09-05 07:59:00.410566: Pseudo dice [0.9584, 0.9517, 0.971, 0.9403, 0.8096, 0.9589, 0.983] 
+2024-09-05 07:59:00.411374: Epoch time: 244.87 s 
+2024-09-05 07:59:02.929405:  
+2024-09-05 07:59:02.930773: Epoch 199 
+2024-09-05 07:59:02.933249: Current learning rate: 0.00819 
+2024-09-05 08:03:41.000070: train_loss -0.6987 
+2024-09-05 08:03:41.002698: val_loss -0.7264 
+2024-09-05 08:03:41.003581: Pseudo dice [0.9583, 0.9512, 0.9729, 0.9364, 0.838, 0.9571, 0.989] 
+2024-09-05 08:03:41.004517: Epoch time: 278.08 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_5_08_31_30.txt

@@ -0,0 +1,380 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-05 08:31:33.825712: Using torch.compile... 
+2024-09-05 08:31:42.566489: do_dummy_2d_data_aug: False 
+2024-09-05 08:31:42.573230: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-05 08:31:42.574775: The split file contains 5 splits. 
+2024-09-05 08:31:42.575571: Desired fold for training: 1 
+2024-09-05 08:31:42.576392: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-05 08:31:56.690818: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-05 08:31:58.406949:  
+2024-09-05 08:31:58.417595: Epoch 200 
+2024-09-05 08:31:58.420135: Current learning rate: 0.00818 
+2024-09-05 08:50:27.953803: train_loss -0.7164 
+2024-09-05 08:50:28.004357: val_loss -0.7345 
+2024-09-05 08:50:28.005304: Pseudo dice [0.9581, 0.9499, 0.9724, 0.9405, 0.8638, 0.9623, 0.9891] 
+2024-09-05 08:50:28.005985: Epoch time: 1109.55 s 
+2024-09-05 08:50:30.463046:  
+2024-09-05 08:50:30.514359: Epoch 201 
+2024-09-05 08:50:30.538584: Current learning rate: 0.00817 
+2024-09-05 08:55:14.354450: train_loss -0.7072 
+2024-09-05 08:55:14.371284: val_loss -0.7171 
+2024-09-05 08:55:14.372291: Pseudo dice [0.9578, 0.9539, 0.9712, 0.9399, 0.8286, 0.9574, 0.9869] 
+2024-09-05 08:55:14.373119: Epoch time: 283.9 s 
+2024-09-05 08:55:17.527075:  
+2024-09-05 08:55:17.528149: Epoch 202 
+2024-09-05 08:55:17.529286: Current learning rate: 0.00816 
+2024-09-05 09:00:04.581638: train_loss -0.7213 
+2024-09-05 09:00:04.593039: val_loss -0.7355 
+2024-09-05 09:00:04.594044: Pseudo dice [0.961, 0.9524, 0.9778, 0.9438, 0.8517, 0.9392, 0.9845] 
+2024-09-05 09:00:04.594943: Epoch time: 287.06 s 
+2024-09-05 09:00:08.144962:  
+2024-09-05 09:00:08.145939: Epoch 203 
+2024-09-05 09:00:08.146784: Current learning rate: 0.00815 
+2024-09-05 09:04:57.866594: train_loss -0.7287 
+2024-09-05 09:04:57.875469: val_loss -0.7133 
+2024-09-05 09:04:57.876445: Pseudo dice [0.9634, 0.9507, 0.9787, 0.95, 0.8597, 0.9632, 0.9882] 
+2024-09-05 09:04:57.877269: Epoch time: 289.73 s 
+2024-09-05 09:05:00.704402:  
+2024-09-05 09:05:00.705713: Epoch 204 
+2024-09-05 09:05:00.706994: Current learning rate: 0.00814 
+2024-09-05 09:09:56.957373: train_loss -0.7389 
+2024-09-05 09:09:56.979114: val_loss -0.748 
+2024-09-05 09:09:56.980083: Pseudo dice [0.9591, 0.9559, 0.9712, 0.9427, 0.8482, 0.9544, 0.9863] 
+2024-09-05 09:09:56.980975: Epoch time: 296.26 s 
+2024-09-05 09:10:01.116535:  
+2024-09-05 09:10:01.117676: Epoch 205 
+2024-09-05 09:10:01.118602: Current learning rate: 0.00813 
+2024-09-05 09:14:57.258225: train_loss -0.7166 
+2024-09-05 09:14:57.274171: val_loss -0.7242 
+2024-09-05 09:14:57.275348: Pseudo dice [0.963, 0.9535, 0.9793, 0.9421, 0.8467, 0.959, 0.9862] 
+2024-09-05 09:14:57.276417: Epoch time: 296.15 s 
+2024-09-05 09:15:01.109316:  
+2024-09-05 09:15:01.110532: Epoch 206 
+2024-09-05 09:15:01.111389: Current learning rate: 0.00813 
+2024-09-05 09:20:15.330712: train_loss -0.7206 
+2024-09-05 09:20:15.332543: val_loss -0.7505 
+2024-09-05 09:20:15.333744: Pseudo dice [0.9621, 0.9544, 0.9781, 0.9434, 0.864, 0.9662, 0.9904] 
+2024-09-05 09:20:15.335139: Epoch time: 314.23 s 
+2024-09-05 09:20:18.195364:  
+2024-09-05 09:20:18.196980: Epoch 207 
+2024-09-05 09:20:18.197921: Current learning rate: 0.00812 
+2024-09-05 09:25:13.518951: train_loss -0.7245 
+2024-09-05 09:25:13.520522: val_loss -0.7351 
+2024-09-05 09:25:13.521447: Pseudo dice [0.958, 0.9541, 0.9743, 0.9444, 0.8653, 0.963, 0.9898] 
+2024-09-05 09:25:13.522278: Epoch time: 295.33 s 
+2024-09-05 09:25:13.523060: Yayy! New best EMA pseudo Dice: 0.9451 
+2024-09-05 09:25:22.060504:  
+2024-09-05 09:25:22.061477: Epoch 208 
+2024-09-05 09:25:22.062378: Current learning rate: 0.00811 
+2024-09-05 09:30:09.777131: train_loss -0.728 
+2024-09-05 09:30:09.793167: val_loss -0.6826 
+2024-09-05 09:30:09.794161: Pseudo dice [0.96, 0.9536, 0.9728, 0.9368, 0.843, 0.9425, 0.9858] 
+2024-09-05 09:30:09.794984: Epoch time: 287.72 s 
+2024-09-05 09:30:12.740765:  
+2024-09-05 09:30:12.741687: Epoch 209 
+2024-09-05 09:30:12.742844: Current learning rate: 0.0081 
+2024-09-05 09:35:08.615083: train_loss -0.6987 
+2024-09-05 09:35:08.631693: val_loss -0.7109 
+2024-09-05 09:35:08.632884: Pseudo dice [0.9562, 0.9455, 0.9717, 0.9387, 0.8281, 0.9609, 0.9871] 
+2024-09-05 09:35:08.634052: Epoch time: 295.88 s 
+2024-09-05 09:35:11.420941:  
+2024-09-05 09:35:11.422269: Epoch 210 
+2024-09-05 09:35:11.423224: Current learning rate: 0.00809 
+2024-09-05 09:40:12.576192: train_loss -0.7119 
+2024-09-05 09:40:12.592174: val_loss -0.7157 
+2024-09-05 09:40:12.593331: Pseudo dice [0.958, 0.9502, 0.9749, 0.9411, 0.848, 0.9513, 0.9863] 
+2024-09-05 09:40:12.594541: Epoch time: 301.16 s 
+2024-09-05 09:40:15.218506:  
+2024-09-05 09:40:15.219687: Epoch 211 
+2024-09-05 09:40:15.220639: Current learning rate: 0.00808 
+2024-09-05 09:45:06.164464: train_loss -0.7177 
+2024-09-05 09:45:06.166766: val_loss -0.7141 
+2024-09-05 09:45:06.167652: Pseudo dice [0.9597, 0.9532, 0.9747, 0.9387, 0.8349, 0.954, 0.9875] 
+2024-09-05 09:45:06.168652: Epoch time: 290.95 s 
+2024-09-05 09:45:09.430495:  
+2024-09-05 09:45:09.431916: Epoch 212 
+2024-09-05 09:45:09.432953: Current learning rate: 0.00807 
+2024-09-05 09:50:09.434871: train_loss -0.7075 
+2024-09-05 09:50:09.436374: val_loss -0.6647 
+2024-09-05 09:50:09.437260: Pseudo dice [0.9511, 0.9475, 0.9665, 0.9274, 0.7931, 0.9293, 0.9858] 
+2024-09-05 09:50:09.438166: Epoch time: 300.01 s 
+2024-09-05 09:50:13.328559:  
+2024-09-05 09:50:13.329783: Epoch 213 
+2024-09-05 09:50:13.330655: Current learning rate: 0.00806 
+2024-09-05 09:55:00.343926: train_loss -0.6967 
+2024-09-05 09:55:00.363862: val_loss -0.6872 
+2024-09-05 09:55:00.365073: Pseudo dice [0.9563, 0.9499, 0.9713, 0.935, 0.8064, 0.9567, 0.9837] 
+2024-09-05 09:55:00.366116: Epoch time: 287.02 s 
+2024-09-05 09:55:03.414621:  
+2024-09-05 09:55:03.415776: Epoch 214 
+2024-09-05 09:55:03.416641: Current learning rate: 0.00805 
+2024-09-05 09:59:55.536554: train_loss -0.7018 
+2024-09-05 09:59:55.538761: val_loss -0.7362 
+2024-09-05 09:59:55.540103: Pseudo dice [0.9579, 0.9486, 0.9757, 0.9434, 0.8158, 0.9628, 0.9856] 
+2024-09-05 09:59:55.541357: Epoch time: 292.13 s 
+2024-09-05 09:59:58.289466:  
+2024-09-05 09:59:58.290562: Epoch 215 
+2024-09-05 09:59:58.291508: Current learning rate: 0.00804 
+2024-09-05 10:05:07.680708: train_loss -0.7209 
+2024-09-05 10:05:07.691942: val_loss -0.7138 
+2024-09-05 10:05:07.693122: Pseudo dice [0.9617, 0.9564, 0.9743, 0.9481, 0.8473, 0.9534, 0.9741] 
+2024-09-05 10:05:07.694228: Epoch time: 309.4 s 
+2024-09-05 10:05:11.861206:  
+2024-09-05 10:05:11.862307: Epoch 216 
+2024-09-05 10:05:11.863110: Current learning rate: 0.00803 
+2024-09-05 10:10:24.528059: train_loss -0.724 
+2024-09-05 10:10:24.543235: val_loss -0.6782 
+2024-09-05 10:10:24.544257: Pseudo dice [0.9511, 0.9524, 0.9688, 0.9241, 0.8409, 0.9629, 0.9878] 
+2024-09-05 10:10:24.545362: Epoch time: 312.67 s 
+2024-09-05 10:10:27.665190:  
+2024-09-05 10:10:27.671084: Epoch 217 
+2024-09-05 10:10:27.672056: Current learning rate: 0.00802 
+2024-09-05 10:15:08.764438: train_loss -0.6897 
+2024-09-05 10:15:08.766568: val_loss -0.6968 
+2024-09-05 10:15:08.767968: Pseudo dice [0.9503, 0.9502, 0.9612, 0.9392, 0.8442, 0.9439, 0.9852] 
+2024-09-05 10:15:08.769272: Epoch time: 281.1 s 
+2024-09-05 10:15:11.945740:  
+2024-09-05 10:15:11.947160: Epoch 218 
+2024-09-05 10:15:11.948276: Current learning rate: 0.00801 
+2024-09-05 10:19:30.835873: train_loss -0.721 
+2024-09-05 10:19:30.837654: val_loss -0.7207 
+2024-09-05 10:19:30.838643: Pseudo dice [0.9558, 0.9503, 0.975, 0.943, 0.8545, 0.9611, 0.9872] 
+2024-09-05 10:19:30.839653: Epoch time: 258.9 s 
+2024-09-05 10:19:33.576518:  
+2024-09-05 10:19:33.577636: Epoch 219 
+2024-09-05 10:19:33.578622: Current learning rate: 0.00801 
+2024-09-05 10:24:09.833591: train_loss -0.7053 
+2024-09-05 10:24:09.847283: val_loss -0.7311 
+2024-09-05 10:24:09.848394: Pseudo dice [0.9601, 0.9536, 0.975, 0.9398, 0.8495, 0.9637, 0.9905] 
+2024-09-05 10:24:09.849282: Epoch time: 276.27 s 
+2024-09-05 10:24:12.400368:  
+2024-09-05 10:24:12.401893: Epoch 220 
+2024-09-05 10:24:12.402894: Current learning rate: 0.008 
+2024-09-05 10:29:09.739615: train_loss -0.709 
+2024-09-05 10:29:09.755261: val_loss -0.7334 
+2024-09-05 10:29:09.756185: Pseudo dice [0.9555, 0.9511, 0.9722, 0.9433, 0.8595, 0.9502, 0.9892] 
+2024-09-05 10:29:09.757094: Epoch time: 297.34 s 
+2024-09-05 10:29:13.947438:  
+2024-09-05 10:29:13.948641: Epoch 221 
+2024-09-05 10:29:13.949693: Current learning rate: 0.00799 
+2024-09-05 10:33:55.368097: train_loss -0.7209 
+2024-09-05 10:33:55.369594: val_loss -0.7178 
+2024-09-05 10:33:55.370583: Pseudo dice [0.9598, 0.9536, 0.9748, 0.9424, 0.8693, 0.9669, 0.9906] 
+2024-09-05 10:33:55.371371: Epoch time: 281.43 s 
+2024-09-05 10:33:59.214845:  
+2024-09-05 10:33:59.216704: Epoch 222 
+2024-09-05 10:33:59.217535: Current learning rate: 0.00798 
+2024-09-05 10:38:52.676672: train_loss -0.7192 
+2024-09-05 10:38:52.688397: val_loss -0.7525 
+2024-09-05 10:38:52.689610: Pseudo dice [0.959, 0.9527, 0.9735, 0.9483, 0.8612, 0.9541, 0.9865] 
+2024-09-05 10:38:52.690793: Epoch time: 293.47 s 
+2024-09-05 10:38:55.355811:  
+2024-09-05 10:38:55.357041: Epoch 223 
+2024-09-05 10:38:55.358004: Current learning rate: 0.00797 
+2024-09-05 10:43:44.708907: train_loss -0.709 
+2024-09-05 10:43:44.723196: val_loss -0.739 
+2024-09-05 10:43:44.724251: Pseudo dice [0.956, 0.9518, 0.9743, 0.9358, 0.8476, 0.9628, 0.9888] 
+2024-09-05 10:43:44.725163: Epoch time: 289.36 s 
+2024-09-05 10:43:47.402081:  
+2024-09-05 10:43:47.403444: Epoch 224 
+2024-09-05 10:43:47.404515: Current learning rate: 0.00796 
+2024-09-05 10:48:27.112138: train_loss -0.7203 
+2024-09-05 10:48:27.114848: val_loss -0.7401 
+2024-09-05 10:48:27.115839: Pseudo dice [0.9609, 0.9556, 0.9752, 0.9407, 0.8712, 0.9573, 0.9879] 
+2024-09-05 10:48:27.116845: Epoch time: 279.71 s 
+2024-09-05 10:48:31.087731:  
+2024-09-05 10:48:31.089050: Epoch 225 
+2024-09-05 10:48:31.089839: Current learning rate: 0.00795 
+2024-09-05 10:53:16.014835: train_loss -0.7283 
+2024-09-05 10:53:16.027961: val_loss -0.7513 
+2024-09-05 10:53:16.028972: Pseudo dice [0.9574, 0.9507, 0.9713, 0.9408, 0.8582, 0.96, 0.9874] 
+2024-09-05 10:53:16.029847: Epoch time: 284.93 s 
+2024-09-05 10:53:16.030667: Yayy! New best EMA pseudo Dice: 0.9452 
+2024-09-05 10:53:24.353070:  
+2024-09-05 10:53:24.353915: Epoch 226 
+2024-09-05 10:53:24.354615: Current learning rate: 0.00794 
+2024-09-05 10:58:03.340238: train_loss -0.724 
+2024-09-05 10:58:03.346715: val_loss -0.746 
+2024-09-05 10:58:03.347703: Pseudo dice [0.9631, 0.9533, 0.9783, 0.9459, 0.8489, 0.9483, 0.9865] 
+2024-09-05 10:58:03.348495: Epoch time: 278.99 s 
+2024-09-05 10:58:03.349249: Yayy! New best EMA pseudo Dice: 0.9453 
+2024-09-05 10:58:11.651080:  
+2024-09-05 10:58:11.652010: Epoch 227 
+2024-09-05 10:58:11.652757: Current learning rate: 0.00793 
+2024-09-05 11:02:54.298186: train_loss -0.7333 
+2024-09-05 11:02:54.302948: val_loss -0.7444 
+2024-09-05 11:02:54.303987: Pseudo dice [0.9582, 0.9516, 0.9726, 0.9453, 0.8707, 0.9637, 0.99] 
+2024-09-05 11:02:54.304856: Epoch time: 282.65 s 
+2024-09-05 11:02:54.305669: Yayy! New best EMA pseudo Dice: 0.9458 
+2024-09-05 11:03:03.319197:  
+2024-09-05 11:03:03.320199: Epoch 228 
+2024-09-05 11:03:03.321137: Current learning rate: 0.00792 
+2024-09-05 11:07:48.871033: train_loss -0.7214 
+2024-09-05 11:07:48.888949: val_loss -0.7474 
+2024-09-05 11:07:48.890823: Pseudo dice [0.9593, 0.9536, 0.9728, 0.9411, 0.8455, 0.9597, 0.9903] 
+2024-09-05 11:07:48.892435: Epoch time: 285.56 s 
+2024-09-05 11:07:48.893731: Yayy! New best EMA pseudo Dice: 0.9458 
+2024-09-05 11:07:57.201512:  
+2024-09-05 11:07:57.202603: Epoch 229 
+2024-09-05 11:07:57.203527: Current learning rate: 0.00791 
+2024-09-05 11:12:27.938564: train_loss -0.7241 
+2024-09-05 11:12:27.940283: val_loss -0.7186 
+2024-09-05 11:12:27.941524: Pseudo dice [0.955, 0.9554, 0.9701, 0.9424, 0.8556, 0.9637, 0.9895] 
+2024-09-05 11:12:27.942621: Epoch time: 270.74 s 
+2024-09-05 11:12:27.943653: Yayy! New best EMA pseudo Dice: 0.946 
+2024-09-05 11:12:35.739894:  
+2024-09-05 11:12:35.740795: Epoch 230 
+2024-09-05 11:12:35.741506: Current learning rate: 0.0079 
+2024-09-05 11:17:19.120358: train_loss -0.7103 
+2024-09-05 11:17:19.142138: val_loss -0.7166 
+2024-09-05 11:17:19.143813: Pseudo dice [0.9557, 0.9513, 0.9714, 0.9347, 0.835, 0.9582, 0.9871] 
+2024-09-05 11:17:19.146966: Epoch time: 283.38 s 
+2024-09-05 11:17:21.924044:  
+2024-09-05 11:17:21.934694: Epoch 231 
+2024-09-05 11:17:21.935803: Current learning rate: 0.00789 
+2024-09-05 11:22:11.012977: train_loss -0.7074 
+2024-09-05 11:22:11.036512: val_loss -0.6848 
+2024-09-05 11:22:11.037540: Pseudo dice [0.9555, 0.9518, 0.969, 0.9338, 0.8271, 0.9453, 0.9851] 
+2024-09-05 11:22:11.038494: Epoch time: 289.09 s 
+2024-09-05 11:22:14.176419:  
+2024-09-05 11:22:14.177469: Epoch 232 
+2024-09-05 11:22:14.178451: Current learning rate: 0.00789 
+2024-09-05 11:27:17.644546: train_loss -0.6933 
+2024-09-05 11:27:17.666308: val_loss -0.7388 
+2024-09-05 11:27:17.667715: Pseudo dice [0.9602, 0.9529, 0.9714, 0.9382, 0.8511, 0.9633, 0.9885] 
+2024-09-05 11:27:17.668873: Epoch time: 303.47 s 
+2024-09-05 11:27:20.690073:  
+2024-09-05 11:27:20.700537: Epoch 233 
+2024-09-05 11:27:20.704432: Current learning rate: 0.00788 
+2024-09-05 11:31:50.336660: train_loss -0.7322 
+2024-09-05 11:31:50.351124: val_loss -0.7215 
+2024-09-05 11:31:50.352525: Pseudo dice [0.9595, 0.9538, 0.9764, 0.941, 0.8357, 0.9572, 0.9857] 
+2024-09-05 11:31:50.353645: Epoch time: 269.65 s 
+2024-09-05 11:31:53.610057:  
+2024-09-05 11:31:53.611147: Epoch 234 
+2024-09-05 11:31:53.611993: Current learning rate: 0.00787 
+2024-09-05 11:36:46.577296: train_loss -0.7208 
+2024-09-05 11:36:46.587635: val_loss -0.7134 
+2024-09-05 11:36:46.594843: Pseudo dice [0.9618, 0.9549, 0.9788, 0.9438, 0.8451, 0.9643, 0.9889] 
+2024-09-05 11:36:46.595840: Epoch time: 292.97 s 
+2024-09-05 11:36:50.256189:  
+2024-09-05 11:36:50.269975: Epoch 235 
+2024-09-05 11:36:50.271227: Current learning rate: 0.00786 
+2024-09-05 11:41:28.734398: train_loss -0.7285 
+2024-09-05 11:41:28.736209: val_loss -0.7189 
+2024-09-05 11:41:28.737324: Pseudo dice [0.9617, 0.9551, 0.9741, 0.9407, 0.8692, 0.967, 0.9909] 
+2024-09-05 11:41:28.738201: Epoch time: 278.48 s 
+2024-09-05 11:41:31.875889:  
+2024-09-05 11:41:31.877490: Epoch 236 
+2024-09-05 11:41:31.878747: Current learning rate: 0.00785 
+2024-09-05 11:46:15.839751: train_loss -0.7252 
+2024-09-05 11:46:15.842577: val_loss -0.7202 
+2024-09-05 11:46:15.844606: Pseudo dice [0.9585, 0.9523, 0.9712, 0.9415, 0.8282, 0.9633, 0.9853] 
+2024-09-05 11:46:15.846509: Epoch time: 283.97 s 
+2024-09-05 11:46:19.423476:  
+2024-09-05 11:46:19.425615: Epoch 237 
+2024-09-05 11:46:19.427062: Current learning rate: 0.00784 
+2024-09-05 11:51:05.669129: train_loss -0.7216 
+2024-09-05 11:51:05.681461: val_loss -0.729 
+2024-09-05 11:51:05.682633: Pseudo dice [0.9622, 0.9509, 0.9777, 0.9423, 0.8305, 0.952, 0.986] 
+2024-09-05 11:51:05.684323: Epoch time: 286.25 s 
+2024-09-05 11:51:08.749205:  
+2024-09-05 11:51:08.754188: Epoch 238 
+2024-09-05 11:51:08.766090: Current learning rate: 0.00783 
+2024-09-05 11:56:05.634062: train_loss -0.7129 
+2024-09-05 11:56:05.637005: val_loss -0.7607 
+2024-09-05 11:56:05.638187: Pseudo dice [0.9609, 0.9498, 0.9756, 0.9433, 0.8358, 0.9618, 0.9852] 
+2024-09-05 11:56:05.639409: Epoch time: 296.89 s 
+2024-09-05 11:56:08.748591:  
+2024-09-05 11:56:08.749625: Epoch 239 
+2024-09-05 11:56:08.750435: Current learning rate: 0.00782 
+2024-09-05 12:01:28.112942: train_loss -0.7168 
+2024-09-05 12:01:28.114818: val_loss -0.7081 
+2024-09-05 12:01:28.116095: Pseudo dice [0.9588, 0.9497, 0.9713, 0.9379, 0.8114, 0.9596, 0.9873] 
+2024-09-05 12:01:28.117162: Epoch time: 319.37 s 
+2024-09-05 12:01:32.934388:  
+2024-09-05 12:01:32.935488: Epoch 240 
+2024-09-05 12:01:32.936501: Current learning rate: 0.00781 
+2024-09-05 12:06:05.727517: train_loss -0.7035 
+2024-09-05 12:06:05.743367: val_loss -0.7201 
+2024-09-05 12:06:05.751552: Pseudo dice [0.96, 0.9539, 0.9724, 0.9456, 0.8187, 0.9627, 0.9862] 
+2024-09-05 12:06:05.752811: Epoch time: 272.8 s 
+2024-09-05 12:06:09.080197:  
+2024-09-05 12:06:09.092918: Epoch 241 
+2024-09-05 12:06:09.094181: Current learning rate: 0.0078 
+2024-09-05 12:10:58.365844: train_loss -0.7268 
+2024-09-05 12:10:58.383943: val_loss -0.7447 
+2024-09-05 12:10:58.385165: Pseudo dice [0.9573, 0.9509, 0.9731, 0.9461, 0.8603, 0.9641, 0.9877] 
+2024-09-05 12:10:58.386081: Epoch time: 289.29 s 
+2024-09-05 12:11:01.979908:  
+2024-09-05 12:11:01.982184: Epoch 242 
+2024-09-05 12:11:01.984314: Current learning rate: 0.00779 
+2024-09-05 12:15:42.470558: train_loss -0.7391 
+2024-09-05 12:15:42.483200: val_loss -0.6967 
+2024-09-05 12:15:42.484180: Pseudo dice [0.9646, 0.9562, 0.9791, 0.9438, 0.8568, 0.9569, 0.9875] 
+2024-09-05 12:15:42.485017: Epoch time: 280.5 s 
+2024-09-05 12:15:46.163462:  
+2024-09-05 12:15:46.174773: Epoch 243 
+2024-09-05 12:15:46.175774: Current learning rate: 0.00778 
+2024-09-05 12:20:43.734342: train_loss -0.7357 
+2024-09-05 12:20:43.747333: val_loss -0.7758 
+2024-09-05 12:20:43.748291: Pseudo dice [0.9613, 0.9538, 0.9757, 0.9469, 0.8707, 0.9506, 0.9894] 
+2024-09-05 12:20:43.749254: Epoch time: 297.58 s 
+2024-09-05 12:20:47.785276:  
+2024-09-05 12:20:47.786606: Epoch 244 
+2024-09-05 12:20:47.787700: Current learning rate: 0.00777 
+2024-09-05 12:25:32.126989: train_loss -0.7246 
+2024-09-05 12:25:32.131009: val_loss -0.7073 
+2024-09-05 12:25:32.132184: Pseudo dice [0.9598, 0.9534, 0.9736, 0.941, 0.8534, 0.9614, 0.9893] 
+2024-09-05 12:25:32.133136: Epoch time: 284.35 s 
+2024-09-05 12:25:35.787421:  
+2024-09-05 12:25:35.788619: Epoch 245 
+2024-09-05 12:25:35.789809: Current learning rate: 0.00777 
+2024-09-05 12:30:17.631586: train_loss -0.712 
+2024-09-05 12:30:17.656488: val_loss -0.7436 
+2024-09-05 12:30:17.657912: Pseudo dice [0.9589, 0.9537, 0.9731, 0.9448, 0.8589, 0.9628, 0.9893] 
+2024-09-05 12:30:17.659005: Epoch time: 281.85 s 
+2024-09-05 12:30:17.660338: Yayy! New best EMA pseudo Dice: 0.9462 
+2024-09-05 12:30:26.152414:  
+2024-09-05 12:30:26.153373: Epoch 246 
+2024-09-05 12:30:26.154335: Current learning rate: 0.00776 
+2024-09-05 12:35:19.366426: train_loss -0.7166 
+2024-09-05 12:35:19.384280: val_loss -0.7171 
+2024-09-05 12:35:19.385658: Pseudo dice [0.9618, 0.954, 0.9758, 0.9402, 0.849, 0.9546, 0.9871] 
+2024-09-05 12:35:19.386699: Epoch time: 293.22 s 
+2024-09-05 12:35:22.901428:  
+2024-09-05 12:35:22.902989: Epoch 247 
+2024-09-05 12:35:22.904460: Current learning rate: 0.00775 
+2024-09-05 12:40:25.823773: train_loss -0.7275 
+2024-09-05 12:40:25.839047: val_loss -0.7171 
+2024-09-05 12:40:25.840401: Pseudo dice [0.9591, 0.9553, 0.9731, 0.9339, 0.849, 0.9529, 0.985] 
+2024-09-05 12:40:25.841591: Epoch time: 302.93 s 
+2024-09-05 12:40:28.438842:  
+2024-09-05 12:40:28.440168: Epoch 248 
+2024-09-05 12:40:28.441172: Current learning rate: 0.00774 
+2024-09-05 12:45:21.532862: train_loss -0.7334 
+2024-09-05 12:45:21.546228: val_loss -0.7515 
+2024-09-05 12:45:21.547747: Pseudo dice [0.9617, 0.9513, 0.9756, 0.9408, 0.8632, 0.9674, 0.9898] 
+2024-09-05 12:45:21.549126: Epoch time: 293.1 s 
+2024-09-05 12:45:21.550408: Yayy! New best EMA pseudo Dice: 0.9464 
+2024-09-05 12:45:31.162979:  
+2024-09-05 12:45:31.163944: Epoch 249 
+2024-09-05 12:45:31.164754: Current learning rate: 0.00773 
+2024-09-05 12:50:00.194288: train_loss -0.7168 
+2024-09-05 12:50:00.268017: val_loss -0.7339 
+2024-09-05 12:50:00.477587: Pseudo dice [0.961, 0.9532, 0.9768, 0.943, 0.861, 0.9663, 0.9902] 
+2024-09-05 12:50:00.558198: Epoch time: 269.03 s 
+2024-09-05 12:50:08.139919: Yayy! New best EMA pseudo Dice: 0.9468 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_5_16_30_44.txt

@@ -0,0 +1,380 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-05 16:30:47.501100: Using torch.compile... 
+2024-09-05 16:30:56.102847: do_dummy_2d_data_aug: False 
+2024-09-05 16:30:56.109239: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-05 16:30:56.111737: The split file contains 5 splits. 
+2024-09-05 16:30:56.112572: Desired fold for training: 1 
+2024-09-05 16:30:56.113266: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-05 16:31:10.166732: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-05 16:31:11.830219:  
+2024-09-05 16:31:11.846095: Epoch 250 
+2024-09-05 16:31:11.848173: Current learning rate: 0.00772 
+2024-09-05 16:49:44.473948: train_loss -0.7112 
+2024-09-05 16:49:44.475238: val_loss -0.7554 
+2024-09-05 16:49:44.475936: Pseudo dice [0.9601, 0.9537, 0.971, 0.9456, 0.8493, 0.9628, 0.989] 
+2024-09-05 16:49:44.476560: Epoch time: 1112.65 s 
+2024-09-05 16:49:44.477160: Yayy! New best EMA pseudo Dice: 0.9468 
+2024-09-05 16:49:57.546696:  
+2024-09-05 16:49:57.547638: Epoch 251 
+2024-09-05 16:49:57.548614: Current learning rate: 0.00771 
+2024-09-05 16:54:19.688939: train_loss -0.7386 
+2024-09-05 16:54:19.690474: val_loss -0.726 
+2024-09-05 16:54:19.692465: Pseudo dice [0.9612, 0.957, 0.9734, 0.9468, 0.8433, 0.9598, 0.983] 
+2024-09-05 16:54:19.694126: Epoch time: 262.15 s 
+2024-09-05 16:54:23.141976:  
+2024-09-05 16:54:23.143144: Epoch 252 
+2024-09-05 16:54:23.144448: Current learning rate: 0.0077 
+2024-09-05 16:58:55.218054: train_loss -0.7265 
+2024-09-05 16:58:55.235441: val_loss -0.7442 
+2024-09-05 16:58:55.236510: Pseudo dice [0.963, 0.9543, 0.9756, 0.9409, 0.8502, 0.9613, 0.9879] 
+2024-09-05 16:58:55.238135: Epoch time: 272.08 s 
+2024-09-05 16:58:55.239020: Yayy! New best EMA pseudo Dice: 0.9469 
+2024-09-05 16:59:05.480499:  
+2024-09-05 16:59:05.481353: Epoch 253 
+2024-09-05 16:59:05.482084: Current learning rate: 0.00769 
+2024-09-05 17:03:29.589779: train_loss -0.7292 
+2024-09-05 17:03:29.607002: val_loss -0.7749 
+2024-09-05 17:03:29.609052: Pseudo dice [0.9616, 0.9561, 0.976, 0.9459, 0.8622, 0.9646, 0.9911] 
+2024-09-05 17:03:29.610291: Epoch time: 264.11 s 
+2024-09-05 17:03:29.611227: Yayy! New best EMA pseudo Dice: 0.9473 
+2024-09-05 17:03:39.101381:  
+2024-09-05 17:03:39.102527: Epoch 254 
+2024-09-05 17:03:39.103298: Current learning rate: 0.00768 
+2024-09-05 17:08:26.841527: train_loss -0.7226 
+2024-09-05 17:08:26.855083: val_loss -0.7506 
+2024-09-05 17:08:26.856436: Pseudo dice [0.9629, 0.9541, 0.9798, 0.9463, 0.8747, 0.9685, 0.9907] 
+2024-09-05 17:08:26.857765: Epoch time: 287.74 s 
+2024-09-05 17:08:26.858772: Yayy! New best EMA pseudo Dice: 0.9479 
+2024-09-05 17:08:40.821829:  
+2024-09-05 17:08:40.832804: Epoch 255 
+2024-09-05 17:08:40.856264: Current learning rate: 0.00767 
+2024-09-05 17:13:21.567890: train_loss -0.7222 
+2024-09-05 17:13:21.569758: val_loss -0.7284 
+2024-09-05 17:13:21.570447: Pseudo dice [0.96, 0.9498, 0.975, 0.9427, 0.8459, 0.9644, 0.9912] 
+2024-09-05 17:13:21.571601: Epoch time: 280.75 s 
+2024-09-05 17:13:25.625668:  
+2024-09-05 17:13:25.626712: Epoch 256 
+2024-09-05 17:13:25.627776: Current learning rate: 0.00766 
+2024-09-05 17:18:02.350731: train_loss -0.7321 
+2024-09-05 17:18:02.369256: val_loss -0.7082 
+2024-09-05 17:18:02.370311: Pseudo dice [0.9591, 0.9533, 0.9707, 0.9467, 0.8429, 0.9573, 0.9877] 
+2024-09-05 17:18:02.371385: Epoch time: 276.73 s 
+2024-09-05 17:18:06.295470:  
+2024-09-05 17:18:06.302672: Epoch 257 
+2024-09-05 17:18:06.303519: Current learning rate: 0.00765 
+2024-09-05 17:22:52.130106: train_loss -0.7274 
+2024-09-05 17:22:52.134175: val_loss -0.7527 
+2024-09-05 17:22:52.135477: Pseudo dice [0.9586, 0.9489, 0.9722, 0.9462, 0.8726, 0.9649, 0.9903] 
+2024-09-05 17:22:52.136623: Epoch time: 285.84 s 
+2024-09-05 17:22:54.617947:  
+2024-09-05 17:22:54.618942: Epoch 258 
+2024-09-05 17:22:54.619626: Current learning rate: 0.00764 
+2024-09-05 17:27:31.115554: train_loss -0.7248 
+2024-09-05 17:27:31.120062: val_loss -0.7272 
+2024-09-05 17:27:31.121069: Pseudo dice [0.9606, 0.9564, 0.9769, 0.9439, 0.86, 0.9636, 0.9902] 
+2024-09-05 17:27:31.122094: Epoch time: 276.5 s 
+2024-09-05 17:27:31.123038: Yayy! New best EMA pseudo Dice: 0.9481 
+2024-09-05 17:27:44.076805:  
+2024-09-05 17:27:44.077666: Epoch 259 
+2024-09-05 17:27:44.078305: Current learning rate: 0.00764 
+2024-09-05 17:32:08.144152: train_loss -0.7374 
+2024-09-05 17:32:08.172084: val_loss -0.7064 
+2024-09-05 17:32:08.172804: Pseudo dice [0.9625, 0.9564, 0.9763, 0.9396, 0.8533, 0.9559, 0.9861] 
+2024-09-05 17:32:08.174030: Epoch time: 264.07 s 
+2024-09-05 17:32:11.010748:  
+2024-09-05 17:32:11.011945: Epoch 260 
+2024-09-05 17:32:11.012888: Current learning rate: 0.00763 
+2024-09-05 17:36:34.796634: train_loss -0.7255 
+2024-09-05 17:36:34.799467: val_loss -0.7362 
+2024-09-05 17:36:34.800291: Pseudo dice [0.9618, 0.9543, 0.9765, 0.9479, 0.8404, 0.9614, 0.9875] 
+2024-09-05 17:36:34.800995: Epoch time: 263.79 s 
+2024-09-05 17:36:36.801501:  
+2024-09-05 17:36:36.802474: Epoch 261 
+2024-09-05 17:36:36.803183: Current learning rate: 0.00762 
+2024-09-05 17:41:22.080729: train_loss -0.7404 
+2024-09-05 17:41:22.098784: val_loss -0.7523 
+2024-09-05 17:41:22.100819: Pseudo dice [0.9633, 0.9551, 0.9775, 0.9482, 0.8285, 0.9603, 0.9864] 
+2024-09-05 17:41:22.101972: Epoch time: 285.28 s 
+2024-09-05 17:41:25.774073:  
+2024-09-05 17:41:25.786953: Epoch 262 
+2024-09-05 17:41:25.788243: Current learning rate: 0.00761 
+2024-09-05 17:45:56.128303: train_loss -0.7382 
+2024-09-05 17:45:56.137326: val_loss -0.7209 
+2024-09-05 17:45:56.138983: Pseudo dice [0.9597, 0.953, 0.974, 0.9482, 0.8701, 0.9605, 0.9907] 
+2024-09-05 17:45:56.140470: Epoch time: 270.36 s 
+2024-09-05 17:45:59.344178:  
+2024-09-05 17:45:59.345308: Epoch 263 
+2024-09-05 17:45:59.346009: Current learning rate: 0.0076 
+2024-09-05 17:50:30.565744: train_loss -0.7225 
+2024-09-05 17:50:30.587035: val_loss -0.7078 
+2024-09-05 17:50:30.588252: Pseudo dice [0.9624, 0.9543, 0.9775, 0.9458, 0.8223, 0.9521, 0.9859] 
+2024-09-05 17:50:30.589157: Epoch time: 271.23 s 
+2024-09-05 17:50:35.626016:  
+2024-09-05 17:50:35.626931: Epoch 264 
+2024-09-05 17:50:35.627636: Current learning rate: 0.00759 
+2024-09-05 17:55:14.808623: train_loss -0.7318 
+2024-09-05 17:55:14.817393: val_loss -0.7257 
+2024-09-05 17:55:14.839207: Pseudo dice [0.9616, 0.9536, 0.9755, 0.9458, 0.8618, 0.9528, 0.9877] 
+2024-09-05 17:55:14.840773: Epoch time: 279.19 s 
+2024-09-05 17:55:17.852928:  
+2024-09-05 17:55:17.854064: Epoch 265 
+2024-09-05 17:55:17.854833: Current learning rate: 0.00758 
+2024-09-05 17:59:59.812349: train_loss -0.7267 
+2024-09-05 17:59:59.829601: val_loss -0.7215 
+2024-09-05 17:59:59.830899: Pseudo dice [0.9628, 0.9556, 0.9789, 0.9483, 0.8617, 0.9655, 0.9901] 
+2024-09-05 17:59:59.832322: Epoch time: 281.96 s 
+2024-09-05 18:00:02.616292:  
+2024-09-05 18:00:02.617643: Epoch 266 
+2024-09-05 18:00:02.629491: Current learning rate: 0.00757 
+2024-09-05 18:04:51.761682: train_loss -0.7264 
+2024-09-05 18:04:51.777611: val_loss -0.7471 
+2024-09-05 18:04:51.778873: Pseudo dice [0.9641, 0.9563, 0.9791, 0.9432, 0.8607, 0.9651, 0.9897] 
+2024-09-05 18:04:51.779981: Epoch time: 289.15 s 
+2024-09-05 18:04:51.780680: Yayy! New best EMA pseudo Dice: 0.9483 
+2024-09-05 18:05:03.184642:  
+2024-09-05 18:05:03.185876: Epoch 267 
+2024-09-05 18:05:03.186595: Current learning rate: 0.00756 
+2024-09-05 18:09:41.688554: train_loss -0.7437 
+2024-09-05 18:09:41.708076: val_loss -0.6994 
+2024-09-05 18:09:41.708829: Pseudo dice [0.9548, 0.9523, 0.9708, 0.9406, 0.8304, 0.9619, 0.9866] 
+2024-09-05 18:09:41.709556: Epoch time: 278.51 s 
+2024-09-05 18:09:44.854420:  
+2024-09-05 18:09:44.855512: Epoch 268 
+2024-09-05 18:09:44.856240: Current learning rate: 0.00755 
+2024-09-05 18:14:07.379044: train_loss -0.7244 
+2024-09-05 18:14:07.395183: val_loss -0.721 
+2024-09-05 18:14:07.396436: Pseudo dice [0.9615, 0.9549, 0.9751, 0.947, 0.868, 0.9534, 0.9888] 
+2024-09-05 18:14:07.397142: Epoch time: 262.53 s 
+2024-09-05 18:14:10.448498:  
+2024-09-05 18:14:10.449551: Epoch 269 
+2024-09-05 18:14:10.450831: Current learning rate: 0.00754 
+2024-09-05 18:18:57.212291: train_loss -0.7051 
+2024-09-05 18:18:57.227618: val_loss -0.7533 
+2024-09-05 18:18:57.229482: Pseudo dice [0.9612, 0.9543, 0.9774, 0.9463, 0.8815, 0.9662, 0.9914] 
+2024-09-05 18:18:57.231198: Epoch time: 286.77 s 
+2024-09-05 18:18:57.233373: Yayy! New best EMA pseudo Dice: 0.9486 
+2024-09-05 18:19:07.910305:  
+2024-09-05 18:19:07.911399: Epoch 270 
+2024-09-05 18:19:07.912152: Current learning rate: 0.00753 
+2024-09-05 18:23:37.065549: train_loss -0.7302 
+2024-09-05 18:23:37.078218: val_loss -0.7177 
+2024-09-05 18:23:37.087042: Pseudo dice [0.9644, 0.9594, 0.9787, 0.9461, 0.8816, 0.9678, 0.9909] 
+2024-09-05 18:23:37.088424: Epoch time: 269.16 s 
+2024-09-05 18:23:37.089788: Yayy! New best EMA pseudo Dice: 0.9493 
+2024-09-05 18:23:47.974558:  
+2024-09-05 18:23:47.975425: Epoch 271 
+2024-09-05 18:23:47.976109: Current learning rate: 0.00752 
+2024-09-05 18:28:14.096804: train_loss -0.7231 
+2024-09-05 18:28:14.111461: val_loss -0.7349 
+2024-09-05 18:28:14.112725: Pseudo dice [0.9628, 0.9545, 0.9786, 0.943, 0.8617, 0.9532, 0.9881] 
+2024-09-05 18:28:14.114328: Epoch time: 266.13 s 
+2024-09-05 18:28:17.021837:  
+2024-09-05 18:28:17.023290: Epoch 272 
+2024-09-05 18:28:17.024195: Current learning rate: 0.00751 
+2024-09-05 18:32:36.620067: train_loss -0.7286 
+2024-09-05 18:32:36.640674: val_loss -0.7517 
+2024-09-05 18:32:36.642277: Pseudo dice [0.9618, 0.954, 0.9789, 0.9461, 0.8725, 0.9602, 0.9881] 
+2024-09-05 18:32:36.645272: Epoch time: 259.6 s 
+2024-09-05 18:32:36.646952: Yayy! New best EMA pseudo Dice: 0.9495 
+2024-09-05 18:32:45.988365:  
+2024-09-05 18:32:45.990104: Epoch 273 
+2024-09-05 18:32:45.990716: Current learning rate: 0.00751 
+2024-09-05 18:37:27.539583: train_loss -0.7322 
+2024-09-05 18:37:27.558803: val_loss -0.7135 
+2024-09-05 18:37:27.559719: Pseudo dice [0.9586, 0.9498, 0.9735, 0.9379, 0.8067, 0.953, 0.9866] 
+2024-09-05 18:37:27.560664: Epoch time: 281.55 s 
+2024-09-05 18:37:30.737687:  
+2024-09-05 18:37:30.738789: Epoch 274 
+2024-09-05 18:37:30.739926: Current learning rate: 0.0075 
+2024-09-05 18:42:08.061331: train_loss -0.7078 
+2024-09-05 18:42:08.079944: val_loss -0.7124 
+2024-09-05 18:42:08.081192: Pseudo dice [0.9555, 0.9517, 0.9731, 0.9378, 0.8382, 0.9607, 0.9883] 
+2024-09-05 18:42:08.082259: Epoch time: 277.33 s 
+2024-09-05 18:42:12.987913:  
+2024-09-05 18:42:12.989039: Epoch 275 
+2024-09-05 18:42:12.989887: Current learning rate: 0.00749 
+2024-09-05 18:46:39.643689: train_loss -0.7134 
+2024-09-05 18:46:39.670806: val_loss -0.7215 
+2024-09-05 18:46:39.671808: Pseudo dice [0.9588, 0.9517, 0.9764, 0.9452, 0.8432, 0.9516, 0.9836] 
+2024-09-05 18:46:39.672585: Epoch time: 266.66 s 
+2024-09-05 18:46:42.021214:  
+2024-09-05 18:46:42.022161: Epoch 276 
+2024-09-05 18:46:42.023163: Current learning rate: 0.00748 
+2024-09-05 18:51:21.135320: train_loss -0.7185 
+2024-09-05 18:51:21.160240: val_loss -0.7382 
+2024-09-05 18:51:21.161637: Pseudo dice [0.9558, 0.9521, 0.9733, 0.935, 0.8437, 0.96, 0.9891] 
+2024-09-05 18:51:21.163069: Epoch time: 279.12 s 
+2024-09-05 18:51:25.009561:  
+2024-09-05 18:51:25.010633: Epoch 277 
+2024-09-05 18:51:25.011495: Current learning rate: 0.00747 
+2024-09-05 18:56:04.389940: train_loss -0.7234 
+2024-09-05 18:56:04.392800: val_loss -0.6943 
+2024-09-05 18:56:04.393951: Pseudo dice [0.958, 0.9509, 0.97, 0.9367, 0.8084, 0.9601, 0.9869] 
+2024-09-05 18:56:04.394796: Epoch time: 279.39 s 
+2024-09-05 18:56:07.152891:  
+2024-09-05 18:56:07.154004: Epoch 278 
+2024-09-05 18:56:07.154806: Current learning rate: 0.00746 
+2024-09-05 19:00:54.239625: train_loss -0.7263 
+2024-09-05 19:00:54.260895: val_loss -0.7674 
+2024-09-05 19:00:54.261768: Pseudo dice [0.9614, 0.9545, 0.9782, 0.942, 0.8601, 0.9622, 0.9877] 
+2024-09-05 19:00:54.262880: Epoch time: 287.09 s 
+2024-09-05 19:00:56.970668:  
+2024-09-05 19:00:56.972254: Epoch 279 
+2024-09-05 19:00:56.973041: Current learning rate: 0.00745 
+2024-09-05 19:05:46.858689: train_loss -0.7136 
+2024-09-05 19:05:46.876893: val_loss -0.6872 
+2024-09-05 19:05:46.879907: Pseudo dice [0.9629, 0.9554, 0.977, 0.949, 0.8702, 0.9283, 0.9829] 
+2024-09-05 19:05:46.880790: Epoch time: 289.89 s 
+2024-09-05 19:05:49.543659:  
+2024-09-05 19:05:49.545276: Epoch 280 
+2024-09-05 19:05:49.546560: Current learning rate: 0.00744 
+2024-09-05 19:10:37.571570: train_loss -0.7377 
+2024-09-05 19:10:37.588516: val_loss -0.7558 
+2024-09-05 19:10:37.589533: Pseudo dice [0.9582, 0.9501, 0.9707, 0.9355, 0.8525, 0.968, 0.9896] 
+2024-09-05 19:10:37.590504: Epoch time: 288.03 s 
+2024-09-05 19:10:40.573905:  
+2024-09-05 19:10:40.574962: Epoch 281 
+2024-09-05 19:10:40.575655: Current learning rate: 0.00743 
+2024-09-05 19:15:35.838412: train_loss -0.7293 
+2024-09-05 19:15:35.850971: val_loss -0.7319 
+2024-09-05 19:15:35.852045: Pseudo dice [0.9581, 0.9531, 0.9725, 0.941, 0.8552, 0.9478, 0.9865] 
+2024-09-05 19:15:35.853343: Epoch time: 295.27 s 
+2024-09-05 19:15:39.810629:  
+2024-09-05 19:15:39.811783: Epoch 282 
+2024-09-05 19:15:39.813122: Current learning rate: 0.00742 
+2024-09-05 19:20:21.387222: train_loss -0.7311 
+2024-09-05 19:20:21.402876: val_loss -0.7325 
+2024-09-05 19:20:21.403725: Pseudo dice [0.9628, 0.9546, 0.979, 0.939, 0.8567, 0.9538, 0.9869] 
+2024-09-05 19:20:21.404453: Epoch time: 281.58 s 
+2024-09-05 19:20:23.622490:  
+2024-09-05 19:20:23.628328: Epoch 283 
+2024-09-05 19:20:23.629449: Current learning rate: 0.00741 
+2024-09-05 19:25:04.671936: train_loss -0.7251 
+2024-09-05 19:25:04.684555: val_loss -0.7469 
+2024-09-05 19:25:04.686148: Pseudo dice [0.9618, 0.9568, 0.9763, 0.9442, 0.861, 0.9604, 0.9904] 
+2024-09-05 19:25:04.689284: Epoch time: 281.06 s 
+2024-09-05 19:25:08.104754:  
+2024-09-05 19:25:08.105775: Epoch 284 
+2024-09-05 19:25:08.106741: Current learning rate: 0.0074 
+2024-09-05 19:30:02.973333: train_loss -0.7327 
+2024-09-05 19:30:03.000656: val_loss -0.718 
+2024-09-05 19:30:03.002429: Pseudo dice [0.9615, 0.9557, 0.9781, 0.9452, 0.8416, 0.9628, 0.9858] 
+2024-09-05 19:30:03.003530: Epoch time: 294.87 s 
+2024-09-05 19:30:05.325366:  
+2024-09-05 19:30:05.326522: Epoch 285 
+2024-09-05 19:30:05.327256: Current learning rate: 0.00739 
+2024-09-05 19:34:54.711669: train_loss -0.7235 
+2024-09-05 19:34:54.722985: val_loss -0.7501 
+2024-09-05 19:34:54.724192: Pseudo dice [0.9655, 0.9589, 0.9799, 0.9496, 0.8764, 0.9669, 0.9905] 
+2024-09-05 19:34:54.725398: Epoch time: 289.39 s 
+2024-09-05 19:34:57.428100:  
+2024-09-05 19:34:57.429184: Epoch 286 
+2024-09-05 19:34:57.430264: Current learning rate: 0.00738 
+2024-09-05 19:39:38.999117: train_loss -0.7272 
+2024-09-05 19:39:39.027936: val_loss -0.7382 
+2024-09-05 19:39:39.030070: Pseudo dice [0.9636, 0.9549, 0.9772, 0.951, 0.8769, 0.9651, 0.9911] 
+2024-09-05 19:39:39.031223: Epoch time: 281.58 s 
+2024-09-05 19:39:42.165237:  
+2024-09-05 19:39:42.166808: Epoch 287 
+2024-09-05 19:39:42.168855: Current learning rate: 0.00738 
+2024-09-05 19:44:26.218091: train_loss -0.7253 
+2024-09-05 19:44:26.235983: val_loss -0.7152 
+2024-09-05 19:44:26.237038: Pseudo dice [0.9626, 0.9551, 0.9758, 0.9502, 0.8744, 0.962, 0.9914] 
+2024-09-05 19:44:26.237982: Epoch time: 284.06 s 
+2024-09-05 19:44:29.004699:  
+2024-09-05 19:44:29.006158: Epoch 288 
+2024-09-05 19:44:29.007044: Current learning rate: 0.00737 
+2024-09-05 19:48:58.347052: train_loss -0.7342 
+2024-09-05 19:48:58.356830: val_loss -0.701 
+2024-09-05 19:48:58.357989: Pseudo dice [0.9604, 0.9564, 0.9746, 0.9447, 0.8319, 0.9561, 0.9845] 
+2024-09-05 19:48:58.358652: Epoch time: 269.35 s 
+2024-09-05 19:49:00.674439:  
+2024-09-05 19:49:00.675686: Epoch 289 
+2024-09-05 19:49:00.676697: Current learning rate: 0.00736 
+2024-09-05 19:53:59.328114: train_loss -0.7377 
+2024-09-05 19:53:59.329747: val_loss -0.7537 
+2024-09-05 19:53:59.331101: Pseudo dice [0.9622, 0.9529, 0.9768, 0.9422, 0.8573, 0.9442, 0.9863] 
+2024-09-05 19:53:59.332084: Epoch time: 298.66 s 
+2024-09-05 19:54:03.457360:  
+2024-09-05 19:54:03.458494: Epoch 290 
+2024-09-05 19:54:03.459824: Current learning rate: 0.00735 
+2024-09-05 19:58:30.289511: train_loss -0.7369 
+2024-09-05 19:58:30.291406: val_loss -0.6841 
+2024-09-05 19:58:30.292700: Pseudo dice [0.9624, 0.9529, 0.9749, 0.9391, 0.781, 0.9644, 0.9805] 
+2024-09-05 19:58:30.294552: Epoch time: 266.84 s 
+2024-09-05 19:58:32.918502:  
+2024-09-05 19:58:32.920567: Epoch 291 
+2024-09-05 19:58:32.921588: Current learning rate: 0.00734 
+2024-09-05 20:03:10.129813: train_loss -0.7232 
+2024-09-05 20:03:10.131941: val_loss -0.7481 
+2024-09-05 20:03:10.133272: Pseudo dice [0.9598, 0.955, 0.9739, 0.9409, 0.8605, 0.9622, 0.9904] 
+2024-09-05 20:03:10.134356: Epoch time: 277.22 s 
+2024-09-05 20:03:14.355944:  
+2024-09-05 20:03:14.357083: Epoch 292 
+2024-09-05 20:03:14.358161: Current learning rate: 0.00733 
+2024-09-05 20:08:01.414275: train_loss -0.7275 
+2024-09-05 20:08:01.427104: val_loss -0.725 
+2024-09-05 20:08:01.428145: Pseudo dice [0.9565, 0.9527, 0.9718, 0.9413, 0.8352, 0.9644, 0.9875] 
+2024-09-05 20:08:01.429024: Epoch time: 287.06 s 
+2024-09-05 20:08:05.073892:  
+2024-09-05 20:08:05.076706: Epoch 293 
+2024-09-05 20:08:05.091072: Current learning rate: 0.00732 
+2024-09-05 20:12:47.522749: train_loss -0.7255 
+2024-09-05 20:12:47.535323: val_loss -0.7704 
+2024-09-05 20:12:47.536798: Pseudo dice [0.9649, 0.956, 0.9796, 0.9503, 0.8787, 0.9668, 0.9914] 
+2024-09-05 20:12:47.539284: Epoch time: 282.45 s 
+2024-09-05 20:12:51.391332:  
+2024-09-05 20:12:51.392383: Epoch 294 
+2024-09-05 20:12:51.393634: Current learning rate: 0.00731 
+2024-09-05 20:17:31.207071: train_loss -0.7323 
+2024-09-05 20:17:31.209042: val_loss -0.7135 
+2024-09-05 20:17:31.210431: Pseudo dice [0.9651, 0.9584, 0.9804, 0.9496, 0.8797, 0.9603, 0.9903] 
+2024-09-05 20:17:31.217326: Epoch time: 279.82 s 
+2024-09-05 20:17:33.534726:  
+2024-09-05 20:17:33.535796: Epoch 295 
+2024-09-05 20:17:33.536586: Current learning rate: 0.0073 
+2024-09-05 20:22:17.373584: train_loss -0.7387 
+2024-09-05 20:22:17.385717: val_loss -0.6884 
+2024-09-05 20:22:17.387286: Pseudo dice [0.9631, 0.9545, 0.9764, 0.9444, 0.8404, 0.9675, 0.9868] 
+2024-09-05 20:22:17.399090: Epoch time: 283.84 s 
+2024-09-05 20:22:20.146827:  
+2024-09-05 20:22:20.147748: Epoch 296 
+2024-09-05 20:22:20.149408: Current learning rate: 0.00729 
+2024-09-05 20:27:09.761381: train_loss -0.7135 
+2024-09-05 20:27:09.782037: val_loss -0.7058 
+2024-09-05 20:27:09.783222: Pseudo dice [0.9607, 0.9555, 0.9755, 0.9439, 0.8691, 0.9583, 0.9883] 
+2024-09-05 20:27:09.784065: Epoch time: 289.62 s 
+2024-09-05 20:27:12.641581:  
+2024-09-05 20:27:12.642658: Epoch 297 
+2024-09-05 20:27:12.643450: Current learning rate: 0.00728 
+2024-09-05 20:31:41.244422: train_loss -0.7142 
+2024-09-05 20:31:41.246903: val_loss -0.7156 
+2024-09-05 20:31:41.247593: Pseudo dice [0.9611, 0.9526, 0.9738, 0.9411, 0.8488, 0.9512, 0.9861] 
+2024-09-05 20:31:41.248227: Epoch time: 268.61 s 
+2024-09-05 20:31:43.656495:  
+2024-09-05 20:31:43.657434: Epoch 298 
+2024-09-05 20:31:43.658146: Current learning rate: 0.00727 
+2024-09-05 20:36:09.031399: train_loss -0.7141 
+2024-09-05 20:36:09.051348: val_loss -0.7385 
+2024-09-05 20:36:09.053831: Pseudo dice [0.9609, 0.9525, 0.9771, 0.9409, 0.8635, 0.9488, 0.986] 
+2024-09-05 20:36:09.054931: Epoch time: 265.38 s 
+2024-09-05 20:36:11.947258:  
+2024-09-05 20:36:11.952237: Epoch 299 
+2024-09-05 20:36:11.953971: Current learning rate: 0.00726 
+2024-09-05 20:41:05.691143: train_loss -0.7428 
+2024-09-05 20:41:05.706790: val_loss -0.6937 
+2024-09-05 20:41:05.708368: Pseudo dice [0.9621, 0.9537, 0.9748, 0.9514, 0.8718, 0.9642, 0.9898] 
+2024-09-05 20:41:05.710289: Epoch time: 293.75 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_6_00_30_15.txt

@@ -0,0 +1,371 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-06 00:30:18.941883: Using torch.compile... 
+2024-09-06 00:30:30.120279: do_dummy_2d_data_aug: False 
+2024-09-06 00:30:30.126404: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-06 00:30:30.128010: The split file contains 5 splits. 
+2024-09-06 00:30:30.128867: Desired fold for training: 1 
+2024-09-06 00:30:30.129645: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-06 00:30:43.950471: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-06 00:30:45.884275:  
+2024-09-06 00:30:45.885711: Epoch 300 
+2024-09-06 00:30:45.887359: Current learning rate: 0.00725 
+2024-09-06 00:49:20.522009: train_loss -0.7209 
+2024-09-06 00:49:20.523568: val_loss -0.7538 
+2024-09-06 00:49:20.524604: Pseudo dice [0.961, 0.9552, 0.9749, 0.9459, 0.8518, 0.9636, 0.9865] 
+2024-09-06 00:49:20.525582: Epoch time: 1114.64 s 
+2024-09-06 00:49:22.606277:  
+2024-09-06 00:49:22.607806: Epoch 301 
+2024-09-06 00:49:22.609293: Current learning rate: 0.00724 
+2024-09-06 00:54:05.755775: train_loss -0.7198 
+2024-09-06 00:54:05.770677: val_loss -0.7464 
+2024-09-06 00:54:05.772932: Pseudo dice [0.9608, 0.9542, 0.9766, 0.9458, 0.8658, 0.9616, 0.9861] 
+2024-09-06 00:54:05.774800: Epoch time: 283.15 s 
+2024-09-06 00:54:09.614479:  
+2024-09-06 00:54:09.625670: Epoch 302 
+2024-09-06 00:54:09.629101: Current learning rate: 0.00724 
+2024-09-06 00:58:55.118804: train_loss -0.7291 
+2024-09-06 00:58:55.133913: val_loss -0.7251 
+2024-09-06 00:58:55.135499: Pseudo dice [0.9625, 0.9569, 0.9781, 0.9388, 0.8562, 0.9649, 0.9881] 
+2024-09-06 00:58:55.138267: Epoch time: 285.51 s 
+2024-09-06 00:58:59.245308:  
+2024-09-06 00:58:59.247244: Epoch 303 
+2024-09-06 00:58:59.249388: Current learning rate: 0.00723 
+2024-09-06 01:03:54.929202: train_loss -0.7355 
+2024-09-06 01:03:54.955788: val_loss -0.7615 
+2024-09-06 01:03:54.957922: Pseudo dice [0.9624, 0.9567, 0.974, 0.9544, 0.8552, 0.9477, 0.9847] 
+2024-09-06 01:03:54.959323: Epoch time: 295.69 s 
+2024-09-06 01:03:59.715424:  
+2024-09-06 01:03:59.882242: Epoch 304 
+2024-09-06 01:03:59.897356: Current learning rate: 0.00722 
+2024-09-06 01:09:10.675407: train_loss -0.7348 
+2024-09-06 01:09:10.678034: val_loss -0.7439 
+2024-09-06 01:09:10.680684: Pseudo dice [0.9635, 0.9578, 0.9783, 0.9445, 0.841, 0.9539, 0.9828] 
+2024-09-06 01:09:10.683003: Epoch time: 310.97 s 
+2024-09-06 01:09:13.921696:  
+2024-09-06 01:09:13.923074: Epoch 305 
+2024-09-06 01:09:13.924228: Current learning rate: 0.00721 
+2024-09-06 01:13:59.470675: train_loss -0.7255 
+2024-09-06 01:13:59.481188: val_loss -0.7501 
+2024-09-06 01:13:59.482258: Pseudo dice [0.9635, 0.9564, 0.9799, 0.943, 0.86, 0.9481, 0.9866] 
+2024-09-06 01:13:59.483495: Epoch time: 285.56 s 
+2024-09-06 01:14:02.419447:  
+2024-09-06 01:14:02.431532: Epoch 306 
+2024-09-06 01:14:02.433697: Current learning rate: 0.0072 
+2024-09-06 01:19:02.194757: train_loss -0.7267 
+2024-09-06 01:19:02.212128: val_loss -0.7434 
+2024-09-06 01:19:02.214402: Pseudo dice [0.9625, 0.9572, 0.9783, 0.9492, 0.8699, 0.951, 0.9888] 
+2024-09-06 01:19:02.215967: Epoch time: 299.78 s 
+2024-09-06 01:19:05.876177:  
+2024-09-06 01:19:05.887079: Epoch 307 
+2024-09-06 01:19:05.888583: Current learning rate: 0.00719 
+2024-09-06 01:24:14.700285: train_loss -0.7302 
+2024-09-06 01:24:14.715981: val_loss -0.7607 
+2024-09-06 01:24:14.717503: Pseudo dice [0.9596, 0.9556, 0.9738, 0.9475, 0.8686, 0.9543, 0.9883] 
+2024-09-06 01:24:14.718855: Epoch time: 308.83 s 
+2024-09-06 01:24:18.018312:  
+2024-09-06 01:24:18.019645: Epoch 308 
+2024-09-06 01:24:18.020775: Current learning rate: 0.00718 
+2024-09-06 01:29:00.847761: train_loss -0.7309 
+2024-09-06 01:29:00.863078: val_loss -0.7154 
+2024-09-06 01:29:00.864280: Pseudo dice [0.9621, 0.9577, 0.9769, 0.9491, 0.8522, 0.9506, 0.9844] 
+2024-09-06 01:29:00.865772: Epoch time: 282.84 s 
+2024-09-06 01:29:04.877289:  
+2024-09-06 01:29:04.878890: Epoch 309 
+2024-09-06 01:29:04.880373: Current learning rate: 0.00717 
+2024-09-06 01:33:45.955226: train_loss -0.7352 
+2024-09-06 01:33:45.958667: val_loss -0.7284 
+2024-09-06 01:33:45.960208: Pseudo dice [0.96, 0.9552, 0.9772, 0.9439, 0.8482, 0.9457, 0.9826] 
+2024-09-06 01:33:45.961718: Epoch time: 281.08 s 
+2024-09-06 01:33:49.296502:  
+2024-09-06 01:33:49.297988: Epoch 310 
+2024-09-06 01:33:49.299231: Current learning rate: 0.00716 
+2024-09-06 01:38:40.151799: train_loss -0.7295 
+2024-09-06 01:38:40.175146: val_loss -0.7448 
+2024-09-06 01:38:40.176567: Pseudo dice [0.9619, 0.9538, 0.9786, 0.9466, 0.8751, 0.9679, 0.9907] 
+2024-09-06 01:38:40.177484: Epoch time: 290.86 s 
+2024-09-06 01:38:43.484946:  
+2024-09-06 01:38:43.493314: Epoch 311 
+2024-09-06 01:38:43.495567: Current learning rate: 0.00715 
+2024-09-06 01:43:44.562276: train_loss -0.7306 
+2024-09-06 01:43:44.575350: val_loss -0.7159 
+2024-09-06 01:43:44.576475: Pseudo dice [0.9593, 0.9537, 0.9755, 0.9431, 0.8616, 0.9657, 0.9883] 
+2024-09-06 01:43:44.577454: Epoch time: 301.08 s 
+2024-09-06 01:43:48.043246:  
+2024-09-06 01:43:48.083405: Epoch 312 
+2024-09-06 01:43:48.324823: Current learning rate: 0.00714 
+2024-09-06 01:48:40.206774: train_loss -0.7287 
+2024-09-06 01:48:40.223713: val_loss -0.7378 
+2024-09-06 01:48:40.225476: Pseudo dice [0.9623, 0.9559, 0.9772, 0.9429, 0.8511, 0.9574, 0.9865] 
+2024-09-06 01:48:40.227518: Epoch time: 292.17 s 
+2024-09-06 01:48:44.020147:  
+2024-09-06 01:48:44.021523: Epoch 313 
+2024-09-06 01:48:44.022612: Current learning rate: 0.00713 
+2024-09-06 01:53:32.129734: train_loss -0.7262 
+2024-09-06 01:53:32.139764: val_loss -0.711 
+2024-09-06 01:53:32.143195: Pseudo dice [0.9642, 0.9555, 0.9786, 0.9465, 0.8534, 0.9635, 0.9888] 
+2024-09-06 01:53:32.144758: Epoch time: 288.12 s 
+2024-09-06 01:53:35.759312:  
+2024-09-06 01:53:35.770837: Epoch 314 
+2024-09-06 01:53:35.772622: Current learning rate: 0.00712 
+2024-09-06 01:58:28.155164: train_loss -0.7256 
+2024-09-06 01:58:28.159862: val_loss -0.7376 
+2024-09-06 01:58:28.161339: Pseudo dice [0.9617, 0.9553, 0.9764, 0.9451, 0.823, 0.9585, 0.9858] 
+2024-09-06 01:58:28.162625: Epoch time: 292.4 s 
+2024-09-06 01:58:31.031976:  
+2024-09-06 01:58:31.033308: Epoch 315 
+2024-09-06 01:58:31.034563: Current learning rate: 0.00711 
+2024-09-06 02:03:13.959616: train_loss -0.7355 
+2024-09-06 02:03:13.961967: val_loss -0.7496 
+2024-09-06 02:03:13.963183: Pseudo dice [0.9608, 0.9525, 0.976, 0.9435, 0.8522, 0.958, 0.9871] 
+2024-09-06 02:03:13.965300: Epoch time: 282.93 s 
+2024-09-06 02:03:17.582888:  
+2024-09-06 02:03:17.594898: Epoch 316 
+2024-09-06 02:03:17.596598: Current learning rate: 0.0071 
+2024-09-06 02:08:27.760520: train_loss -0.7312 
+2024-09-06 02:08:27.763051: val_loss -0.7608 
+2024-09-06 02:08:27.764408: Pseudo dice [0.961, 0.9546, 0.9753, 0.9441, 0.8734, 0.9672, 0.9896] 
+2024-09-06 02:08:27.765624: Epoch time: 310.18 s 
+2024-09-06 02:08:30.835538:  
+2024-09-06 02:08:30.836675: Epoch 317 
+2024-09-06 02:08:30.837537: Current learning rate: 0.0071 
+2024-09-06 02:13:26.292201: train_loss -0.734 
+2024-09-06 02:13:26.296823: val_loss -0.7329 
+2024-09-06 02:13:26.299602: Pseudo dice [0.9589, 0.9571, 0.9722, 0.9459, 0.8767, 0.9683, 0.9918] 
+2024-09-06 02:13:26.301488: Epoch time: 295.46 s 
+2024-09-06 02:13:30.440337:  
+2024-09-06 02:13:30.441720: Epoch 318 
+2024-09-06 02:13:30.443355: Current learning rate: 0.00709 
+2024-09-06 02:18:42.645640: train_loss -0.7259 
+2024-09-06 02:18:42.657907: val_loss -0.7149 
+2024-09-06 02:18:42.659800: Pseudo dice [0.962, 0.9566, 0.9736, 0.9481, 0.8667, 0.9516, 0.9876] 
+2024-09-06 02:18:42.661431: Epoch time: 312.21 s 
+2024-09-06 02:18:45.795168:  
+2024-09-06 02:18:45.796621: Epoch 319 
+2024-09-06 02:18:45.797852: Current learning rate: 0.00708 
+2024-09-06 02:23:36.208740: train_loss -0.7331 
+2024-09-06 02:23:36.219032: val_loss -0.7585 
+2024-09-06 02:23:36.220232: Pseudo dice [0.9613, 0.9535, 0.9766, 0.9422, 0.8582, 0.9653, 0.9894] 
+2024-09-06 02:23:36.221378: Epoch time: 290.42 s 
+2024-09-06 02:23:39.385369:  
+2024-09-06 02:23:39.386981: Epoch 320 
+2024-09-06 02:23:39.388596: Current learning rate: 0.00707 
+2024-09-06 02:28:42.456412: train_loss -0.7181 
+2024-09-06 02:28:42.467588: val_loss -0.7394 
+2024-09-06 02:28:42.469019: Pseudo dice [0.9633, 0.9526, 0.9786, 0.9472, 0.8364, 0.9491, 0.9832] 
+2024-09-06 02:28:42.470305: Epoch time: 303.08 s 
+2024-09-06 02:28:45.823186:  
+2024-09-06 02:28:45.824720: Epoch 321 
+2024-09-06 02:28:45.826298: Current learning rate: 0.00706 
+2024-09-06 02:33:42.964424: train_loss -0.7117 
+2024-09-06 02:33:42.979256: val_loss -0.7093 
+2024-09-06 02:33:42.980611: Pseudo dice [0.9575, 0.9491, 0.9743, 0.9415, 0.8632, 0.9489, 0.9843] 
+2024-09-06 02:33:42.981710: Epoch time: 297.15 s 
+2024-09-06 02:33:45.848534:  
+2024-09-06 02:33:45.862921: Epoch 322 
+2024-09-06 02:33:45.864307: Current learning rate: 0.00705 
+2024-09-06 02:38:37.645555: train_loss -0.7164 
+2024-09-06 02:38:37.647649: val_loss -0.7572 
+2024-09-06 02:38:37.649065: Pseudo dice [0.9632, 0.9538, 0.9784, 0.9438, 0.8584, 0.9625, 0.989] 
+2024-09-06 02:38:37.650226: Epoch time: 291.81 s 
+2024-09-06 02:38:41.336379:  
+2024-09-06 02:38:41.346853: Epoch 323 
+2024-09-06 02:38:41.348101: Current learning rate: 0.00704 
+2024-09-06 02:43:34.326158: train_loss -0.7169 
+2024-09-06 02:43:34.335663: val_loss -0.7192 
+2024-09-06 02:43:34.337236: Pseudo dice [0.9609, 0.9533, 0.9762, 0.9435, 0.8486, 0.9392, 0.9828] 
+2024-09-06 02:43:34.338809: Epoch time: 293.0 s 
+2024-09-06 02:43:37.135121:  
+2024-09-06 02:43:37.137222: Epoch 324 
+2024-09-06 02:43:37.139503: Current learning rate: 0.00703 
+2024-09-06 02:48:33.247695: train_loss -0.7347 
+2024-09-06 02:48:33.259495: val_loss -0.7239 
+2024-09-06 02:48:33.261237: Pseudo dice [0.9604, 0.9553, 0.974, 0.9434, 0.8362, 0.9664, 0.9856] 
+2024-09-06 02:48:33.263816: Epoch time: 296.12 s 
+2024-09-06 02:48:37.899799:  
+2024-09-06 02:48:37.901567: Epoch 325 
+2024-09-06 02:48:37.902846: Current learning rate: 0.00702 
+2024-09-06 02:53:27.367983: train_loss -0.7352 
+2024-09-06 02:53:27.378951: val_loss -0.7146 
+2024-09-06 02:53:27.380508: Pseudo dice [0.961, 0.956, 0.9763, 0.9416, 0.8532, 0.9641, 0.9868] 
+2024-09-06 02:53:27.382380: Epoch time: 289.47 s 
+2024-09-06 02:53:31.938049:  
+2024-09-06 02:53:31.939560: Epoch 326 
+2024-09-06 02:53:31.940664: Current learning rate: 0.00701 
+2024-09-06 02:58:30.268272: train_loss -0.7253 
+2024-09-06 02:58:30.283571: val_loss -0.7306 
+2024-09-06 02:58:30.284677: Pseudo dice [0.9606, 0.9541, 0.9756, 0.9484, 0.8339, 0.96, 0.9843] 
+2024-09-06 02:58:30.299117: Epoch time: 298.33 s 
+2024-09-06 02:58:34.117348:  
+2024-09-06 02:58:34.119769: Epoch 327 
+2024-09-06 02:58:34.121121: Current learning rate: 0.007 
+2024-09-06 03:03:36.627390: train_loss -0.731 
+2024-09-06 03:03:36.643997: val_loss -0.6724 
+2024-09-06 03:03:36.645772: Pseudo dice [0.9639, 0.9571, 0.9777, 0.9469, 0.8321, 0.9632, 0.9863] 
+2024-09-06 03:03:36.647254: Epoch time: 302.52 s 
+2024-09-06 03:03:39.475609:  
+2024-09-06 03:03:39.476804: Epoch 328 
+2024-09-06 03:03:39.478113: Current learning rate: 0.00699 
+2024-09-06 03:08:30.172147: train_loss -0.7345 
+2024-09-06 03:08:30.187993: val_loss -0.743 
+2024-09-06 03:08:30.199418: Pseudo dice [0.964, 0.9556, 0.9783, 0.9473, 0.8816, 0.9404, 0.9854] 
+2024-09-06 03:08:30.201170: Epoch time: 290.7 s 
+2024-09-06 03:08:32.819110:  
+2024-09-06 03:08:32.822574: Epoch 329 
+2024-09-06 03:08:32.823839: Current learning rate: 0.00698 
+2024-09-06 03:13:22.553358: train_loss -0.7342 
+2024-09-06 03:13:22.572113: val_loss -0.7027 
+2024-09-06 03:13:22.573826: Pseudo dice [0.9643, 0.9583, 0.9795, 0.9462, 0.8074, 0.9459, 0.9812] 
+2024-09-06 03:13:22.575031: Epoch time: 289.74 s 
+2024-09-06 03:13:25.444438:  
+2024-09-06 03:13:25.445884: Epoch 330 
+2024-09-06 03:13:25.446909: Current learning rate: 0.00697 
+2024-09-06 03:18:19.391366: train_loss -0.7402 
+2024-09-06 03:18:19.407364: val_loss -0.7448 
+2024-09-06 03:18:19.408545: Pseudo dice [0.9625, 0.9597, 0.9788, 0.9412, 0.8365, 0.953, 0.9811] 
+2024-09-06 03:18:19.409479: Epoch time: 293.97 s 
+2024-09-06 03:18:22.576971:  
+2024-09-06 03:18:22.578332: Epoch 331 
+2024-09-06 03:18:22.579573: Current learning rate: 0.00696 
+2024-09-06 03:23:22.677944: train_loss -0.7245 
+2024-09-06 03:23:22.680842: val_loss -0.7525 
+2024-09-06 03:23:22.682757: Pseudo dice [0.9619, 0.9562, 0.9777, 0.9422, 0.8789, 0.9685, 0.9893] 
+2024-09-06 03:23:22.684632: Epoch time: 300.11 s 
+2024-09-06 03:23:25.570875:  
+2024-09-06 03:23:25.575344: Epoch 332 
+2024-09-06 03:23:25.582378: Current learning rate: 0.00696 
+2024-09-06 03:28:11.489895: train_loss -0.7338 
+2024-09-06 03:28:11.515478: val_loss -0.7213 
+2024-09-06 03:28:11.523664: Pseudo dice [0.9637, 0.9558, 0.9784, 0.944, 0.8391, 0.9644, 0.9855] 
+2024-09-06 03:28:11.526047: Epoch time: 285.92 s 
+2024-09-06 03:28:14.789893:  
+2024-09-06 03:28:14.791456: Epoch 333 
+2024-09-06 03:28:14.794143: Current learning rate: 0.00695 
+2024-09-06 03:33:06.500146: train_loss -0.7101 
+2024-09-06 03:33:06.507489: val_loss -0.6934 
+2024-09-06 03:33:06.509506: Pseudo dice [0.9541, 0.9525, 0.9649, 0.923, 0.8561, 0.9663, 0.99] 
+2024-09-06 03:33:06.510882: Epoch time: 291.72 s 
+2024-09-06 03:33:09.713635:  
+2024-09-06 03:33:09.727430: Epoch 334 
+2024-09-06 03:33:09.729836: Current learning rate: 0.00694 
+2024-09-06 03:38:05.270776: train_loss -0.7223 
+2024-09-06 03:38:05.275131: val_loss -0.7414 
+2024-09-06 03:38:05.290349: Pseudo dice [0.9622, 0.9557, 0.9751, 0.9439, 0.848, 0.9607, 0.9863] 
+2024-09-06 03:38:05.291923: Epoch time: 295.56 s 
+2024-09-06 03:38:08.032239:  
+2024-09-06 03:38:08.033529: Epoch 335 
+2024-09-06 03:38:08.034876: Current learning rate: 0.00693 
+2024-09-06 03:43:00.801107: train_loss -0.7199 
+2024-09-06 03:43:00.828011: val_loss -0.7256 
+2024-09-06 03:43:00.829677: Pseudo dice [0.9581, 0.9519, 0.9699, 0.9418, 0.8639, 0.9543, 0.9877] 
+2024-09-06 03:43:00.831417: Epoch time: 292.77 s 
+2024-09-06 03:43:04.311705:  
+2024-09-06 03:43:04.326969: Epoch 336 
+2024-09-06 03:43:04.329255: Current learning rate: 0.00692 
+2024-09-06 03:48:16.295691: train_loss -0.6967 
+2024-09-06 03:48:16.298478: val_loss -0.6745 
+2024-09-06 03:48:16.300527: Pseudo dice [0.9502, 0.9424, 0.966, 0.9336, 0.8112, 0.9566, 0.9826] 
+2024-09-06 03:48:16.303621: Epoch time: 311.99 s 
+2024-09-06 03:48:19.990195:  
+2024-09-06 03:48:19.991621: Epoch 337 
+2024-09-06 03:48:19.992983: Current learning rate: 0.00691 
+2024-09-06 03:53:06.991296: train_loss -0.6973 
+2024-09-06 03:53:06.996852: val_loss -0.7046 
+2024-09-06 03:53:06.999031: Pseudo dice [0.9567, 0.9488, 0.9671, 0.9353, 0.8518, 0.9583, 0.9887] 
+2024-09-06 03:53:07.001413: Epoch time: 287.01 s 
+2024-09-06 03:53:10.920349:  
+2024-09-06 03:53:10.921526: Epoch 338 
+2024-09-06 03:53:10.922908: Current learning rate: 0.0069 
+2024-09-06 03:58:10.669721: train_loss -0.694 
+2024-09-06 03:58:10.672786: val_loss -0.7069 
+2024-09-06 03:58:10.673967: Pseudo dice [0.9511, 0.9495, 0.9676, 0.9453, 0.8607, 0.9513, 0.9855] 
+2024-09-06 03:58:10.674926: Epoch time: 299.75 s 
+2024-09-06 03:58:14.172410:  
+2024-09-06 03:58:14.173630: Epoch 339 
+2024-09-06 03:58:14.174754: Current learning rate: 0.00689 
+2024-09-06 04:03:02.360500: train_loss -0.7051 
+2024-09-06 04:03:02.362987: val_loss -0.6979 
+2024-09-06 04:03:02.364269: Pseudo dice [0.9565, 0.9524, 0.97, 0.9446, 0.8133, 0.9624, 0.9848] 
+2024-09-06 04:03:02.366428: Epoch time: 288.19 s 
+2024-09-06 04:03:05.042458:  
+2024-09-06 04:03:05.044165: Epoch 340 
+2024-09-06 04:03:05.045372: Current learning rate: 0.00688 
+2024-09-06 04:07:50.748454: train_loss -0.7233 
+2024-09-06 04:07:50.767525: val_loss -0.7693 
+2024-09-06 04:07:50.768807: Pseudo dice [0.9595, 0.9515, 0.9752, 0.9396, 0.866, 0.9653, 0.9914] 
+2024-09-06 04:07:50.769924: Epoch time: 285.71 s 
+2024-09-06 04:07:54.152903:  
+2024-09-06 04:07:54.166990: Epoch 341 
+2024-09-06 04:07:54.168497: Current learning rate: 0.00687 
+2024-09-06 04:12:42.126843: train_loss -0.7168 
+2024-09-06 04:12:42.128729: val_loss -0.7246 
+2024-09-06 04:12:42.130022: Pseudo dice [0.9619, 0.9545, 0.9742, 0.9444, 0.8607, 0.9536, 0.9875] 
+2024-09-06 04:12:42.131737: Epoch time: 287.98 s 
+2024-09-06 04:12:45.162790:  
+2024-09-06 04:12:45.164021: Epoch 342 
+2024-09-06 04:12:45.165160: Current learning rate: 0.00686 
+2024-09-06 04:17:30.481417: train_loss -0.7211 
+2024-09-06 04:17:30.510863: val_loss -0.7413 
+2024-09-06 04:17:30.512592: Pseudo dice [0.965, 0.9553, 0.9805, 0.9493, 0.8601, 0.9655, 0.9874] 
+2024-09-06 04:17:30.514143: Epoch time: 285.32 s 
+2024-09-06 04:17:34.478215:  
+2024-09-06 04:17:34.479672: Epoch 343 
+2024-09-06 04:17:34.481792: Current learning rate: 0.00685 
+2024-09-06 04:22:11.497254: train_loss -0.7224 
+2024-09-06 04:22:11.520312: val_loss -0.7352 
+2024-09-06 04:22:11.522054: Pseudo dice [0.9604, 0.9531, 0.9746, 0.9396, 0.835, 0.9656, 0.9894] 
+2024-09-06 04:22:11.523405: Epoch time: 277.03 s 
+2024-09-06 04:22:15.899872:  
+2024-09-06 04:22:15.901562: Epoch 344 
+2024-09-06 04:22:15.902907: Current learning rate: 0.00684 
+2024-09-06 04:27:03.509989: train_loss -0.7278 
+2024-09-06 04:27:03.527405: val_loss -0.7274 
+2024-09-06 04:27:03.528441: Pseudo dice [0.9586, 0.953, 0.9706, 0.9463, 0.8519, 0.9461, 0.9863] 
+2024-09-06 04:27:03.529411: Epoch time: 287.62 s 
+2024-09-06 04:27:06.129685:  
+2024-09-06 04:27:06.131283: Epoch 345 
+2024-09-06 04:27:06.132353: Current learning rate: 0.00683 
+2024-09-06 04:32:02.808479: train_loss -0.7106 
+2024-09-06 04:32:02.823579: val_loss -0.746 
+2024-09-06 04:32:02.824592: Pseudo dice [0.9614, 0.9534, 0.9755, 0.9417, 0.8692, 0.9668, 0.9916] 
+2024-09-06 04:32:02.826001: Epoch time: 296.68 s 
+2024-09-06 04:32:05.603723:  
+2024-09-06 04:32:05.605526: Epoch 346 
+2024-09-06 04:32:05.607323: Current learning rate: 0.00682 
+2024-09-06 04:37:01.023230: train_loss -0.721 
+2024-09-06 04:37:01.043474: val_loss -0.7483 
+2024-09-06 04:37:01.044737: Pseudo dice [0.9631, 0.9573, 0.9794, 0.9461, 0.8255, 0.965, 0.9857] 
+2024-09-06 04:37:01.045914: Epoch time: 295.42 s 
+2024-09-06 04:37:04.735427:  
+2024-09-06 04:37:04.736970: Epoch 347 
+2024-09-06 04:37:04.737926: Current learning rate: 0.00681 
+2024-09-06 04:41:54.846451: train_loss -0.7226 
+2024-09-06 04:41:54.852011: val_loss -0.7494 
+2024-09-06 04:41:54.853085: Pseudo dice [0.9608, 0.954, 0.9778, 0.9499, 0.862, 0.9687, 0.9877] 
+2024-09-06 04:41:54.854256: Epoch time: 290.12 s 
+2024-09-06 04:41:58.672594:  
+2024-09-06 04:41:58.674183: Epoch 348 
+2024-09-06 04:41:58.675673: Current learning rate: 0.0068 
+2024-09-06 04:47:02.840694: train_loss -0.7256 
+2024-09-06 04:47:02.850189: val_loss -0.7428 
+2024-09-06 04:47:02.853023: Pseudo dice [0.9599, 0.9546, 0.9739, 0.9461, 0.8694, 0.9655, 0.9889] 
+2024-09-06 04:47:02.855072: Epoch time: 304.17 s 
+2024-09-06 04:47:06.757977:  
+2024-09-06 04:47:06.759236: Epoch 349 
+2024-09-06 04:47:06.760936: Current learning rate: 0.0068 
+2024-09-06 04:51:51.725698: train_loss -0.7313 
+2024-09-06 04:51:51.735869: val_loss -0.7465 
+2024-09-06 04:51:51.737767: Pseudo dice [0.9621, 0.955, 0.9763, 0.9425, 0.8597, 0.967, 0.989] 
+2024-09-06 04:51:51.739540: Epoch time: 284.97 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_6_08_29_45.txt

@@ -0,0 +1,382 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-06 08:29:48.103674: Using torch.compile... 
+2024-09-06 08:29:56.209432: do_dummy_2d_data_aug: False 
+2024-09-06 08:29:56.216229: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-06 08:29:56.218817: The split file contains 5 splits. 
+2024-09-06 08:29:56.219441: Desired fold for training: 1 
+2024-09-06 08:29:56.220096: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-06 08:30:10.130210: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-06 08:30:11.906002:  
+2024-09-06 08:30:11.906985: Epoch 350 
+2024-09-06 08:30:11.908757: Current learning rate: 0.00679 
+2024-09-06 08:48:47.074912: train_loss -0.7379 
+2024-09-06 08:48:47.076344: val_loss -0.716 
+2024-09-06 08:48:47.077098: Pseudo dice [0.9604, 0.9543, 0.9756, 0.9428, 0.8523, 0.943, 0.9841] 
+2024-09-06 08:48:47.077811: Epoch time: 1115.18 s 
+2024-09-06 08:48:48.883025:  
+2024-09-06 08:48:48.884074: Epoch 351 
+2024-09-06 08:48:48.884713: Current learning rate: 0.00678 
+2024-09-06 08:53:19.406682: train_loss -0.7392 
+2024-09-06 08:53:19.422330: val_loss -0.734 
+2024-09-06 08:53:19.423275: Pseudo dice [0.9625, 0.9577, 0.9775, 0.9506, 0.8501, 0.9661, 0.9873] 
+2024-09-06 08:53:19.426159: Epoch time: 270.53 s 
+2024-09-06 08:53:21.957958:  
+2024-09-06 08:53:21.959732: Epoch 352 
+2024-09-06 08:53:21.961427: Current learning rate: 0.00677 
+2024-09-06 08:57:56.972915: train_loss -0.7119 
+2024-09-06 08:57:56.986759: val_loss -0.7055 
+2024-09-06 08:57:56.988198: Pseudo dice [0.9632, 0.9545, 0.9753, 0.9419, 0.8362, 0.9524, 0.9875] 
+2024-09-06 08:57:56.989229: Epoch time: 275.02 s 
+2024-09-06 08:58:01.078187:  
+2024-09-06 08:58:01.079324: Epoch 353 
+2024-09-06 08:58:01.080467: Current learning rate: 0.00676 
+2024-09-06 09:02:36.613803: train_loss -0.7431 
+2024-09-06 09:02:36.630949: val_loss -0.7598 
+2024-09-06 09:02:36.631930: Pseudo dice [0.9648, 0.9564, 0.9783, 0.9476, 0.871, 0.9656, 0.9901] 
+2024-09-06 09:02:36.632768: Epoch time: 275.54 s 
+2024-09-06 09:02:40.735322:  
+2024-09-06 09:02:40.736558: Epoch 354 
+2024-09-06 09:02:40.737519: Current learning rate: 0.00675 
+2024-09-06 09:07:31.956928: train_loss -0.7307 
+2024-09-06 09:07:31.958641: val_loss -0.7376 
+2024-09-06 09:07:31.959455: Pseudo dice [0.9594, 0.956, 0.9713, 0.9404, 0.8528, 0.9639, 0.9879] 
+2024-09-06 09:07:31.960214: Epoch time: 291.23 s 
+2024-09-06 09:07:34.387265:  
+2024-09-06 09:07:34.389180: Epoch 355 
+2024-09-06 09:07:34.398815: Current learning rate: 0.00674 
+2024-09-06 09:12:22.053252: train_loss -0.7331 
+2024-09-06 09:12:22.059973: val_loss -0.7455 
+2024-09-06 09:12:22.060854: Pseudo dice [0.9634, 0.9567, 0.9785, 0.9465, 0.8377, 0.9633, 0.9861] 
+2024-09-06 09:12:22.062194: Epoch time: 287.67 s 
+2024-09-06 09:12:25.375845:  
+2024-09-06 09:12:25.377037: Epoch 356 
+2024-09-06 09:12:25.380638: Current learning rate: 0.00673 
+2024-09-06 09:17:19.470980: train_loss -0.739 
+2024-09-06 09:17:19.481537: val_loss -0.7267 
+2024-09-06 09:17:19.482968: Pseudo dice [0.9636, 0.9557, 0.9794, 0.9461, 0.853, 0.969, 0.9896] 
+2024-09-06 09:17:19.484096: Epoch time: 294.1 s 
+2024-09-06 09:17:22.841408:  
+2024-09-06 09:17:22.855376: Epoch 357 
+2024-09-06 09:17:22.856488: Current learning rate: 0.00672 
+2024-09-06 09:22:11.593046: train_loss -0.738 
+2024-09-06 09:22:11.606914: val_loss -0.739 
+2024-09-06 09:22:11.607924: Pseudo dice [0.9615, 0.9539, 0.9769, 0.9494, 0.8679, 0.9685, 0.9897] 
+2024-09-06 09:22:11.608946: Epoch time: 288.76 s 
+2024-09-06 09:22:14.959300:  
+2024-09-06 09:22:14.960329: Epoch 358 
+2024-09-06 09:22:14.961164: Current learning rate: 0.00671 
+2024-09-06 09:27:00.870107: train_loss -0.7419 
+2024-09-06 09:27:00.883203: val_loss -0.7446 
+2024-09-06 09:27:00.884263: Pseudo dice [0.9623, 0.959, 0.9763, 0.9517, 0.8634, 0.9604, 0.9861] 
+2024-09-06 09:27:00.884963: Epoch time: 285.92 s 
+2024-09-06 09:27:03.919564:  
+2024-09-06 09:27:03.920681: Epoch 359 
+2024-09-06 09:27:03.921508: Current learning rate: 0.0067 
+2024-09-06 09:31:55.940557: train_loss -0.7418 
+2024-09-06 09:31:55.955081: val_loss -0.746 
+2024-09-06 09:31:55.955925: Pseudo dice [0.9623, 0.9541, 0.9773, 0.9467, 0.8575, 0.9697, 0.9892] 
+2024-09-06 09:31:55.956668: Epoch time: 292.03 s 
+2024-09-06 09:31:58.822868:  
+2024-09-06 09:31:58.824053: Epoch 360 
+2024-09-06 09:31:58.825361: Current learning rate: 0.00669 
+2024-09-06 09:36:37.652502: train_loss -0.7415 
+2024-09-06 09:36:37.660807: val_loss -0.7462 
+2024-09-06 09:36:37.661793: Pseudo dice [0.9619, 0.9533, 0.9761, 0.9397, 0.8666, 0.9677, 0.9904] 
+2024-09-06 09:36:37.662727: Epoch time: 278.83 s 
+2024-09-06 09:36:40.898276:  
+2024-09-06 09:36:40.899295: Epoch 361 
+2024-09-06 09:36:40.900289: Current learning rate: 0.00668 
+2024-09-06 09:41:24.631229: train_loss -0.7274 
+2024-09-06 09:41:24.645689: val_loss -0.7404 
+2024-09-06 09:41:24.647165: Pseudo dice [0.9629, 0.9575, 0.976, 0.9483, 0.8473, 0.9655, 0.986] 
+2024-09-06 09:41:24.649223: Epoch time: 283.74 s 
+2024-09-06 09:41:28.307224:  
+2024-09-06 09:41:28.308457: Epoch 362 
+2024-09-06 09:41:28.309120: Current learning rate: 0.00667 
+2024-09-06 09:46:00.644626: train_loss -0.7368 
+2024-09-06 09:46:00.655749: val_loss -0.7522 
+2024-09-06 09:46:00.656810: Pseudo dice [0.9596, 0.9558, 0.9747, 0.9457, 0.8596, 0.9639, 0.9888] 
+2024-09-06 09:46:00.658134: Epoch time: 272.34 s 
+2024-09-06 09:46:04.750879:  
+2024-09-06 09:46:04.752047: Epoch 363 
+2024-09-06 09:46:04.753166: Current learning rate: 0.00666 
+2024-09-06 09:50:50.980735: train_loss -0.7417 
+2024-09-06 09:50:51.023040: val_loss -0.7725 
+2024-09-06 09:50:51.024075: Pseudo dice [0.9635, 0.9541, 0.9764, 0.9472, 0.8576, 0.965, 0.9887] 
+2024-09-06 09:50:51.024995: Epoch time: 286.24 s 
+2024-09-06 09:50:54.811031:  
+2024-09-06 09:50:54.814706: Epoch 364 
+2024-09-06 09:50:54.815690: Current learning rate: 0.00665 
+2024-09-06 09:55:24.233462: train_loss -0.7346 
+2024-09-06 09:55:24.236019: val_loss -0.752 
+2024-09-06 09:55:24.237427: Pseudo dice [0.964, 0.9551, 0.9786, 0.9472, 0.8549, 0.9698, 0.9884] 
+2024-09-06 09:55:24.241967: Epoch time: 269.43 s 
+2024-09-06 09:55:24.254973: Yayy! New best EMA pseudo Dice: 0.9496 
+2024-09-06 09:55:34.285617:  
+2024-09-06 09:55:34.286776: Epoch 365 
+2024-09-06 09:55:34.287688: Current learning rate: 0.00665 
+2024-09-06 10:00:13.660185: train_loss -0.7332 
+2024-09-06 10:00:13.678910: val_loss -0.7421 
+2024-09-06 10:00:13.680530: Pseudo dice [0.9629, 0.9554, 0.9761, 0.9443, 0.8508, 0.9684, 0.9897] 
+2024-09-06 10:00:13.681679: Epoch time: 279.38 s 
+2024-09-06 10:00:13.683065: Yayy! New best EMA pseudo Dice: 0.9496 
+2024-09-06 10:00:24.062034:  
+2024-09-06 10:00:24.063039: Epoch 366 
+2024-09-06 10:00:24.063760: Current learning rate: 0.00664 
+2024-09-06 10:04:59.181149: train_loss -0.7498 
+2024-09-06 10:04:59.191600: val_loss -0.7713 
+2024-09-06 10:04:59.193262: Pseudo dice [0.9615, 0.9565, 0.9751, 0.9447, 0.867, 0.963, 0.9886] 
+2024-09-06 10:04:59.194625: Epoch time: 275.12 s 
+2024-09-06 10:04:59.195580: Yayy! New best EMA pseudo Dice: 0.9497 
+2024-09-06 10:05:08.143336:  
+2024-09-06 10:05:08.144245: Epoch 367 
+2024-09-06 10:05:08.145068: Current learning rate: 0.00663 
+2024-09-06 10:09:50.338075: train_loss -0.7453 
+2024-09-06 10:09:50.351639: val_loss -0.7351 
+2024-09-06 10:09:50.352785: Pseudo dice [0.9594, 0.957, 0.9709, 0.9471, 0.8662, 0.9582, 0.988] 
+2024-09-06 10:09:50.353772: Epoch time: 282.2 s 
+2024-09-06 10:09:53.452492:  
+2024-09-06 10:09:53.453703: Epoch 368 
+2024-09-06 10:09:53.454741: Current learning rate: 0.00662 
+2024-09-06 10:14:19.158549: train_loss -0.7435 
+2024-09-06 10:14:19.171047: val_loss -0.7499 
+2024-09-06 10:14:19.172161: Pseudo dice [0.9633, 0.9559, 0.9776, 0.9479, 0.851, 0.9673, 0.9874] 
+2024-09-06 10:14:19.173136: Epoch time: 265.71 s 
+2024-09-06 10:14:19.174173: Yayy! New best EMA pseudo Dice: 0.9497 
+2024-09-06 10:14:29.117687:  
+2024-09-06 10:14:29.118726: Epoch 369 
+2024-09-06 10:14:29.119481: Current learning rate: 0.00661 
+2024-09-06 10:19:09.782426: train_loss -0.7204 
+2024-09-06 10:19:09.791437: val_loss -0.7445 
+2024-09-06 10:19:09.792581: Pseudo dice [0.9642, 0.9563, 0.9781, 0.9485, 0.8633, 0.97, 0.988] 
+2024-09-06 10:19:09.793625: Epoch time: 280.67 s 
+2024-09-06 10:19:09.794932: Yayy! New best EMA pseudo Dice: 0.95 
+2024-09-06 10:19:17.483839:  
+2024-09-06 10:19:17.484969: Epoch 370 
+2024-09-06 10:19:17.485551: Current learning rate: 0.0066 
+2024-09-06 10:24:12.003322: train_loss -0.7372 
+2024-09-06 10:24:12.005392: val_loss -0.7169 
+2024-09-06 10:24:12.006414: Pseudo dice [0.9633, 0.9557, 0.978, 0.9448, 0.8536, 0.9683, 0.9901] 
+2024-09-06 10:24:12.007324: Epoch time: 294.52 s 
+2024-09-06 10:24:12.008378: Yayy! New best EMA pseudo Dice: 0.9501 
+2024-09-06 10:24:21.180121:  
+2024-09-06 10:24:21.181088: Epoch 371 
+2024-09-06 10:24:21.181656: Current learning rate: 0.00659 
+2024-09-06 10:29:15.942341: train_loss -0.7297 
+2024-09-06 10:29:15.944020: val_loss -0.7512 
+2024-09-06 10:29:15.944737: Pseudo dice [0.9644, 0.956, 0.9786, 0.9495, 0.8424, 0.9609, 0.9855] 
+2024-09-06 10:29:15.945843: Epoch time: 294.77 s 
+2024-09-06 10:29:19.476502:  
+2024-09-06 10:29:19.477952: Epoch 372 
+2024-09-06 10:29:19.480187: Current learning rate: 0.00658 
+2024-09-06 10:34:02.127073: train_loss -0.7344 
+2024-09-06 10:34:02.143734: val_loss -0.7402 
+2024-09-06 10:34:02.145015: Pseudo dice [0.9612, 0.9533, 0.9771, 0.9439, 0.857, 0.9563, 0.9874] 
+2024-09-06 10:34:02.148026: Epoch time: 282.66 s 
+2024-09-06 10:34:05.165494:  
+2024-09-06 10:34:05.168141: Epoch 373 
+2024-09-06 10:34:05.171957: Current learning rate: 0.00657 
+2024-09-06 10:38:48.806200: train_loss -0.7333 
+2024-09-06 10:38:48.831187: val_loss -0.7707 
+2024-09-06 10:38:48.832561: Pseudo dice [0.9643, 0.9556, 0.9794, 0.9495, 0.866, 0.9663, 0.9881] 
+2024-09-06 10:38:48.833624: Epoch time: 283.65 s 
+2024-09-06 10:38:53.291646:  
+2024-09-06 10:38:53.292791: Epoch 374 
+2024-09-06 10:38:53.293633: Current learning rate: 0.00656 
+2024-09-06 10:44:04.449541: train_loss -0.7294 
+2024-09-06 10:44:04.462859: val_loss -0.7216 
+2024-09-06 10:44:04.463723: Pseudo dice [0.959, 0.9545, 0.9742, 0.942, 0.8667, 0.9513, 0.9867] 
+2024-09-06 10:44:04.464424: Epoch time: 311.16 s 
+2024-09-06 10:44:07.158694:  
+2024-09-06 10:44:07.160118: Epoch 375 
+2024-09-06 10:44:07.161054: Current learning rate: 0.00655 
+2024-09-06 10:48:58.950650: train_loss -0.7252 
+2024-09-06 10:48:58.963125: val_loss -0.7701 
+2024-09-06 10:48:58.964037: Pseudo dice [0.9621, 0.9545, 0.9786, 0.9447, 0.8812, 0.9699, 0.9921] 
+2024-09-06 10:48:58.964983: Epoch time: 291.8 s 
+2024-09-06 10:48:58.965937: Yayy! New best EMA pseudo Dice: 0.9503 
+2024-09-06 10:49:11.742391:  
+2024-09-06 10:49:11.744159: Epoch 376 
+2024-09-06 10:49:11.744949: Current learning rate: 0.00654 
+2024-09-06 10:53:51.862494: train_loss -0.7132 
+2024-09-06 10:53:51.867667: val_loss -0.7025 
+2024-09-06 10:53:51.869374: Pseudo dice [0.9526, 0.9493, 0.9693, 0.936, 0.852, 0.9502, 0.9839] 
+2024-09-06 10:53:51.870995: Epoch time: 280.12 s 
+2024-09-06 10:53:54.260594:  
+2024-09-06 10:53:54.261732: Epoch 377 
+2024-09-06 10:53:54.262452: Current learning rate: 0.00653 
+2024-09-06 10:58:48.220788: train_loss -0.7101 
+2024-09-06 10:58:48.236174: val_loss -0.7342 
+2024-09-06 10:58:48.236964: Pseudo dice [0.9602, 0.955, 0.9735, 0.9437, 0.852, 0.9651, 0.9885] 
+2024-09-06 10:58:48.238006: Epoch time: 293.97 s 
+2024-09-06 10:58:51.143979:  
+2024-09-06 10:58:51.145549: Epoch 378 
+2024-09-06 10:58:51.146994: Current learning rate: 0.00652 
+2024-09-06 11:03:32.105277: train_loss -0.7247 
+2024-09-06 11:03:32.119680: val_loss -0.7178 
+2024-09-06 11:03:32.120666: Pseudo dice [0.959, 0.9572, 0.9737, 0.9507, 0.8642, 0.9664, 0.9899] 
+2024-09-06 11:03:32.121378: Epoch time: 280.97 s 
+2024-09-06 11:03:35.217400:  
+2024-09-06 11:03:35.218814: Epoch 379 
+2024-09-06 11:03:35.219897: Current learning rate: 0.00651 
+2024-09-06 11:08:21.170814: train_loss -0.723 
+2024-09-06 11:08:21.191164: val_loss -0.7807 
+2024-09-06 11:08:21.192144: Pseudo dice [0.9606, 0.9542, 0.9726, 0.9486, 0.8799, 0.9687, 0.9918] 
+2024-09-06 11:08:21.192922: Epoch time: 285.96 s 
+2024-09-06 11:08:24.342794:  
+2024-09-06 11:08:24.346228: Epoch 380 
+2024-09-06 11:08:24.347435: Current learning rate: 0.0065 
+2024-09-06 11:13:07.850982: train_loss -0.7297 
+2024-09-06 11:13:07.869084: val_loss -0.7396 
+2024-09-06 11:13:07.870213: Pseudo dice [0.9593, 0.9546, 0.9756, 0.9401, 0.8681, 0.9628, 0.9899] 
+2024-09-06 11:13:07.871475: Epoch time: 283.51 s 
+2024-09-06 11:13:11.186252:  
+2024-09-06 11:13:11.187594: Epoch 381 
+2024-09-06 11:13:11.188755: Current learning rate: 0.00649 
+2024-09-06 11:18:06.545371: train_loss -0.7283 
+2024-09-06 11:18:06.559165: val_loss -0.7415 
+2024-09-06 11:18:06.559986: Pseudo dice [0.9608, 0.9537, 0.9734, 0.9439, 0.8507, 0.9651, 0.9884] 
+2024-09-06 11:18:06.561041: Epoch time: 295.36 s 
+2024-09-06 11:18:10.077229:  
+2024-09-06 11:18:10.090817: Epoch 382 
+2024-09-06 11:18:10.091919: Current learning rate: 0.00648 
+2024-09-06 11:23:18.356697: train_loss -0.725 
+2024-09-06 11:23:18.374021: val_loss -0.774 
+2024-09-06 11:23:18.375465: Pseudo dice [0.9653, 0.9554, 0.9785, 0.9473, 0.8662, 0.9651, 0.9897] 
+2024-09-06 11:23:18.376621: Epoch time: 308.29 s 
+2024-09-06 11:23:21.852401:  
+2024-09-06 11:23:21.853691: Epoch 383 
+2024-09-06 11:23:21.854772: Current learning rate: 0.00648 
+2024-09-06 11:28:04.409747: train_loss -0.7367 
+2024-09-06 11:28:04.416933: val_loss -0.7299 
+2024-09-06 11:28:04.417846: Pseudo dice [0.9659, 0.9602, 0.978, 0.9506, 0.8835, 0.9682, 0.9906] 
+2024-09-06 11:28:04.419455: Epoch time: 282.56 s 
+2024-09-06 11:28:04.420533: Yayy! New best EMA pseudo Dice: 0.9507 
+2024-09-06 11:28:14.216035:  
+2024-09-06 11:28:14.216952: Epoch 384 
+2024-09-06 11:28:14.217642: Current learning rate: 0.00647 
+2024-09-06 11:33:14.095424: train_loss -0.7198 
+2024-09-06 11:33:14.108788: val_loss -0.7093 
+2024-09-06 11:33:14.110525: Pseudo dice [0.9663, 0.9564, 0.9803, 0.9506, 0.8506, 0.9655, 0.9882] 
+2024-09-06 11:33:14.111628: Epoch time: 299.88 s 
+2024-09-06 11:33:14.112557: Yayy! New best EMA pseudo Dice: 0.9508 
+2024-09-06 11:33:24.086188:  
+2024-09-06 11:33:24.087140: Epoch 385 
+2024-09-06 11:33:24.087868: Current learning rate: 0.00646 
+2024-09-06 11:38:35.226732: train_loss -0.733 
+2024-09-06 11:38:35.248819: val_loss -0.7551 
+2024-09-06 11:38:35.250482: Pseudo dice [0.9618, 0.9542, 0.9777, 0.9443, 0.8681, 0.9661, 0.9902] 
+2024-09-06 11:38:35.252128: Epoch time: 311.14 s 
+2024-09-06 11:38:35.253244: Yayy! New best EMA pseudo Dice: 0.9509 
+2024-09-06 11:38:45.194410:  
+2024-09-06 11:38:45.195509: Epoch 386 
+2024-09-06 11:38:45.196394: Current learning rate: 0.00645 
+2024-09-06 11:43:35.927369: train_loss -0.7376 
+2024-09-06 11:43:35.940130: val_loss -0.7457 
+2024-09-06 11:43:35.941713: Pseudo dice [0.9632, 0.9572, 0.9795, 0.9487, 0.8468, 0.9681, 0.9863] 
+2024-09-06 11:43:35.942920: Epoch time: 290.74 s 
+2024-09-06 11:43:39.415270:  
+2024-09-06 11:43:39.416399: Epoch 387 
+2024-09-06 11:43:39.417109: Current learning rate: 0.00644 
+2024-09-06 11:48:04.193703: train_loss -0.7531 
+2024-09-06 11:48:04.208260: val_loss -0.7227 
+2024-09-06 11:48:04.209031: Pseudo dice [0.9636, 0.9563, 0.9779, 0.9539, 0.8885, 0.9683, 0.992] 
+2024-09-06 11:48:04.209912: Epoch time: 264.78 s 
+2024-09-06 11:48:04.210876: Yayy! New best EMA pseudo Dice: 0.9514 
+2024-09-06 11:48:14.027666:  
+2024-09-06 11:48:14.028934: Epoch 388 
+2024-09-06 11:48:14.029974: Current learning rate: 0.00643 
+2024-09-06 11:52:51.734327: train_loss -0.7315 
+2024-09-06 11:52:51.753295: val_loss -0.7245 
+2024-09-06 11:52:51.756803: Pseudo dice [0.9582, 0.9515, 0.9741, 0.9362, 0.8592, 0.9585, 0.99] 
+2024-09-06 11:52:51.758563: Epoch time: 277.71 s 
+2024-09-06 11:52:55.168216:  
+2024-09-06 11:52:55.169655: Epoch 389 
+2024-09-06 11:52:55.170723: Current learning rate: 0.00642 
+2024-09-06 11:57:32.425533: train_loss -0.7429 
+2024-09-06 11:57:32.444756: val_loss -0.7661 
+2024-09-06 11:57:32.445616: Pseudo dice [0.9597, 0.9556, 0.9743, 0.9456, 0.8568, 0.9687, 0.9888] 
+2024-09-06 11:57:32.447735: Epoch time: 277.26 s 
+2024-09-06 11:57:35.644283:  
+2024-09-06 11:57:35.645361: Epoch 390 
+2024-09-06 11:57:35.646531: Current learning rate: 0.00641 
+2024-09-06 12:02:15.990442: train_loss -0.7137 
+2024-09-06 12:02:16.009212: val_loss -0.7661 
+2024-09-06 12:02:16.011115: Pseudo dice [0.9602, 0.9552, 0.9759, 0.9465, 0.8634, 0.9656, 0.9907] 
+2024-09-06 12:02:16.012635: Epoch time: 280.35 s 
+2024-09-06 12:02:19.277959:  
+2024-09-06 12:02:19.279305: Epoch 391 
+2024-09-06 12:02:19.280135: Current learning rate: 0.0064 
+2024-09-06 12:07:14.088805: train_loss -0.7303 
+2024-09-06 12:07:14.109235: val_loss -0.7103 
+2024-09-06 12:07:14.110594: Pseudo dice [0.9627, 0.9562, 0.9785, 0.9483, 0.8616, 0.9514, 0.9857] 
+2024-09-06 12:07:14.111759: Epoch time: 294.82 s 
+2024-09-06 12:07:17.992329:  
+2024-09-06 12:07:17.993301: Epoch 392 
+2024-09-06 12:07:17.994167: Current learning rate: 0.00639 
+2024-09-06 12:12:14.088682: train_loss -0.7463 
+2024-09-06 12:12:14.104335: val_loss -0.7457 
+2024-09-06 12:12:14.105422: Pseudo dice [0.9637, 0.9565, 0.9786, 0.9443, 0.8662, 0.9654, 0.9889] 
+2024-09-06 12:12:14.106763: Epoch time: 296.1 s 
+2024-09-06 12:12:17.567632:  
+2024-09-06 12:12:17.568741: Epoch 393 
+2024-09-06 12:12:17.569573: Current learning rate: 0.00638 
+2024-09-06 12:16:57.636936: train_loss -0.7205 
+2024-09-06 12:16:57.660451: val_loss -0.7303 
+2024-09-06 12:16:57.662096: Pseudo dice [0.9595, 0.954, 0.9748, 0.9421, 0.8603, 0.9474, 0.9859] 
+2024-09-06 12:16:57.663033: Epoch time: 280.08 s 
+2024-09-06 12:17:00.483908:  
+2024-09-06 12:17:00.485980: Epoch 394 
+2024-09-06 12:17:00.487252: Current learning rate: 0.00637 
+2024-09-06 12:21:37.891836: train_loss -0.7286 
+2024-09-06 12:21:37.908675: val_loss -0.728 
+2024-09-06 12:21:37.910094: Pseudo dice [0.9608, 0.9531, 0.9766, 0.9314, 0.8393, 0.9584, 0.989] 
+2024-09-06 12:21:37.910865: Epoch time: 277.41 s 
+2024-09-06 12:21:42.807360:  
+2024-09-06 12:21:42.814832: Epoch 395 
+2024-09-06 12:21:42.817587: Current learning rate: 0.00636 
+2024-09-06 12:26:14.613429: train_loss -0.7324 
+2024-09-06 12:26:14.631275: val_loss -0.7591 
+2024-09-06 12:26:14.632809: Pseudo dice [0.9635, 0.9579, 0.9773, 0.9469, 0.8404, 0.9649, 0.986] 
+2024-09-06 12:26:14.634526: Epoch time: 271.81 s 
+2024-09-06 12:26:17.652535:  
+2024-09-06 12:26:17.653744: Epoch 396 
+2024-09-06 12:26:17.654428: Current learning rate: 0.00635 
+2024-09-06 12:31:00.962514: train_loss -0.724 
+2024-09-06 12:31:00.964587: val_loss -0.7135 
+2024-09-06 12:31:00.966193: Pseudo dice [0.9636, 0.9545, 0.9761, 0.947, 0.8305, 0.9686, 0.9851] 
+2024-09-06 12:31:00.967592: Epoch time: 283.32 s 
+2024-09-06 12:31:03.950299:  
+2024-09-06 12:31:03.951891: Epoch 397 
+2024-09-06 12:31:03.953142: Current learning rate: 0.00634 
+2024-09-06 12:35:53.142778: train_loss -0.7321 
+2024-09-06 12:35:53.159268: val_loss -0.732 
+2024-09-06 12:35:53.160575: Pseudo dice [0.9571, 0.9539, 0.9683, 0.936, 0.856, 0.9636, 0.9887] 
+2024-09-06 12:35:53.161469: Epoch time: 289.2 s 
+2024-09-06 12:35:57.006634:  
+2024-09-06 12:35:57.007956: Epoch 398 
+2024-09-06 12:35:57.008940: Current learning rate: 0.00633 
+2024-09-06 12:40:37.573433: train_loss -0.7366 
+2024-09-06 12:40:37.595457: val_loss -0.7479 
+2024-09-06 12:40:37.603186: Pseudo dice [0.9581, 0.9494, 0.9689, 0.9429, 0.8484, 0.9673, 0.9892] 
+2024-09-06 12:40:37.604399: Epoch time: 280.57 s 
+2024-09-06 12:40:41.700734:  
+2024-09-06 12:40:41.702115: Epoch 399 
+2024-09-06 12:40:41.702921: Current learning rate: 0.00632 
+2024-09-06 12:45:24.561687: train_loss -0.7137 
+2024-09-06 12:45:24.563180: val_loss -0.7292 
+2024-09-06 12:45:24.564435: Pseudo dice [0.9529, 0.9484, 0.9708, 0.9321, 0.8428, 0.9581, 0.9888] 
+2024-09-06 12:45:24.565773: Epoch time: 282.87 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_6_17_12_15.txt

@@ -0,0 +1,371 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-06 17:12:18.640642: Using torch.compile... 
+2024-09-06 17:12:24.492839: do_dummy_2d_data_aug: False 
+2024-09-06 17:12:24.504089: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-06 17:12:24.518806: The split file contains 5 splits. 
+2024-09-06 17:12:24.519246: Desired fold for training: 1 
+2024-09-06 17:12:24.519669: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-06 17:12:40.725603: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-06 17:12:41.784923:  
+2024-09-06 17:12:41.785813: Epoch 400 
+2024-09-06 17:12:41.786529: Current learning rate: 0.00631 
+2024-09-06 17:27:25.474089: train_loss -0.7024 
+2024-09-06 17:27:25.476377: val_loss -0.7687 
+2024-09-06 17:27:25.476799: Pseudo dice [0.9605, 0.9515, 0.9713, 0.9424, 0.8582, 0.9646, 0.9891] 
+2024-09-06 17:27:25.477174: Epoch time: 883.69 s 
+2024-09-06 17:27:27.056375:  
+2024-09-06 17:27:27.057071: Epoch 401 
+2024-09-06 17:27:27.057580: Current learning rate: 0.0063 
+2024-09-06 17:35:13.643147: train_loss -0.7312 
+2024-09-06 17:35:13.656501: val_loss -0.7299 
+2024-09-06 17:35:13.657221: Pseudo dice [0.9645, 0.9545, 0.9779, 0.9477, 0.8615, 0.9534, 0.9885] 
+2024-09-06 17:35:13.657808: Epoch time: 466.59 s 
+2024-09-06 17:35:17.032650:  
+2024-09-06 17:35:17.033617: Epoch 402 
+2024-09-06 17:35:17.035744: Current learning rate: 0.0063 
+2024-09-06 17:42:50.418836: train_loss -0.7386 
+2024-09-06 17:42:50.422960: val_loss -0.7397 
+2024-09-06 17:42:50.423609: Pseudo dice [0.9627, 0.9561, 0.977, 0.9482, 0.8698, 0.9684, 0.9904] 
+2024-09-06 17:42:50.424043: Epoch time: 453.39 s 
+2024-09-06 17:42:54.697127:  
+2024-09-06 17:42:54.697871: Epoch 403 
+2024-09-06 17:42:54.698562: Current learning rate: 0.00629 
+2024-09-06 17:49:18.405139: train_loss -0.7361 
+2024-09-06 17:49:18.415253: val_loss -0.7656 
+2024-09-06 17:49:18.415852: Pseudo dice [0.9651, 0.9574, 0.9802, 0.9528, 0.8432, 0.9673, 0.9877] 
+2024-09-06 17:49:18.416373: Epoch time: 383.71 s 
+2024-09-06 17:49:21.674520:  
+2024-09-06 17:49:21.685134: Epoch 404 
+2024-09-06 17:49:21.696264: Current learning rate: 0.00628 
+2024-09-06 17:56:04.035905: train_loss -0.751 
+2024-09-06 17:56:04.039096: val_loss -0.7359 
+2024-09-06 17:56:04.039982: Pseudo dice [0.9612, 0.953, 0.9738, 0.9446, 0.8628, 0.9661, 0.9895] 
+2024-09-06 17:56:04.040512: Epoch time: 402.37 s 
+2024-09-06 17:56:07.729840:  
+2024-09-06 17:56:07.730535: Epoch 405 
+2024-09-06 17:56:07.731085: Current learning rate: 0.00627 
+2024-09-06 18:03:01.028334: train_loss -0.7417 
+2024-09-06 18:03:01.031118: val_loss -0.7533 
+2024-09-06 18:03:01.031927: Pseudo dice [0.9636, 0.9566, 0.9772, 0.9486, 0.8652, 0.9479, 0.9856] 
+2024-09-06 18:03:01.032688: Epoch time: 413.3 s 
+2024-09-06 18:03:05.363978:  
+2024-09-06 18:03:05.365708: Epoch 406 
+2024-09-06 18:03:05.366650: Current learning rate: 0.00626 
+2024-09-06 18:09:42.696064: train_loss -0.7388 
+2024-09-06 18:09:42.741408: val_loss -0.732 
+2024-09-06 18:09:42.742010: Pseudo dice [0.9639, 0.9559, 0.9775, 0.947, 0.8878, 0.9682, 0.992] 
+2024-09-06 18:09:42.742564: Epoch time: 397.36 s 
+2024-09-06 18:09:45.385661:  
+2024-09-06 18:09:45.386774: Epoch 407 
+2024-09-06 18:09:45.387564: Current learning rate: 0.00625 
+2024-09-06 18:16:10.537705: train_loss -0.739 
+2024-09-06 18:16:10.550797: val_loss -0.738 
+2024-09-06 18:16:10.551314: Pseudo dice [0.9642, 0.9584, 0.9783, 0.9521, 0.862, 0.9526, 0.9845] 
+2024-09-06 18:16:10.551830: Epoch time: 385.16 s 
+2024-09-06 18:16:13.070060:  
+2024-09-06 18:16:13.082893: Epoch 408 
+2024-09-06 18:16:13.086282: Current learning rate: 0.00624 
+2024-09-06 18:22:11.094954: train_loss -0.7162 
+2024-09-06 18:22:11.097720: val_loss -0.743 
+2024-09-06 18:22:11.098245: Pseudo dice [0.9577, 0.9508, 0.971, 0.9495, 0.8699, 0.9551, 0.9886] 
+2024-09-06 18:22:11.098652: Epoch time: 358.05 s 
+2024-09-06 18:22:14.183839:  
+2024-09-06 18:22:14.184865: Epoch 409 
+2024-09-06 18:22:14.186183: Current learning rate: 0.00623 
+2024-09-06 18:28:36.419572: train_loss -0.7364 
+2024-09-06 18:28:36.429886: val_loss -0.7762 
+2024-09-06 18:28:36.430471: Pseudo dice [0.9611, 0.954, 0.9774, 0.9432, 0.8732, 0.9701, 0.9911] 
+2024-09-06 18:28:36.431188: Epoch time: 382.24 s 
+2024-09-06 18:28:38.490380:  
+2024-09-06 18:28:38.491103: Epoch 410 
+2024-09-06 18:28:38.491668: Current learning rate: 0.00622 
+2024-09-06 18:35:31.522377: train_loss -0.7388 
+2024-09-06 18:35:31.525028: val_loss -0.7342 
+2024-09-06 18:35:31.525496: Pseudo dice [0.9557, 0.9541, 0.9692, 0.9459, 0.8826, 0.9694, 0.9911] 
+2024-09-06 18:35:31.525925: Epoch time: 413.04 s 
+2024-09-06 18:35:33.790506:  
+2024-09-06 18:35:33.791207: Epoch 411 
+2024-09-06 18:35:33.791734: Current learning rate: 0.00621 
+2024-09-06 18:41:41.850688: train_loss -0.7451 
+2024-09-06 18:41:41.874255: val_loss -0.737 
+2024-09-06 18:41:41.874846: Pseudo dice [0.9626, 0.9576, 0.9754, 0.9529, 0.8666, 0.9687, 0.9898] 
+2024-09-06 18:41:41.875315: Epoch time: 368.06 s 
+2024-09-06 18:41:46.353563:  
+2024-09-06 18:41:46.355417: Epoch 412 
+2024-09-06 18:41:46.356445: Current learning rate: 0.0062 
+2024-09-06 18:47:32.269139: train_loss -0.7286 
+2024-09-06 18:47:32.281071: val_loss -0.7345 
+2024-09-06 18:47:32.281629: Pseudo dice [0.9629, 0.9527, 0.9774, 0.9376, 0.8618, 0.9428, 0.988] 
+2024-09-06 18:47:32.282122: Epoch time: 345.93 s 
+2024-09-06 18:47:35.604229:  
+2024-09-06 18:47:35.605670: Epoch 413 
+2024-09-06 18:47:35.606375: Current learning rate: 0.00619 
+2024-09-06 18:53:18.179193: train_loss -0.7204 
+2024-09-06 18:53:18.190120: val_loss -0.7209 
+2024-09-06 18:53:18.190689: Pseudo dice [0.966, 0.9572, 0.9783, 0.947, 0.86, 0.9566, 0.9868] 
+2024-09-06 18:53:18.191306: Epoch time: 342.58 s 
+2024-09-06 18:53:21.377557:  
+2024-09-06 18:53:21.393129: Epoch 414 
+2024-09-06 18:53:21.394852: Current learning rate: 0.00618 
+2024-09-06 18:59:07.656974: train_loss -0.7309 
+2024-09-06 18:59:07.662637: val_loss -0.7345 
+2024-09-06 18:59:07.663747: Pseudo dice [0.9626, 0.9557, 0.9777, 0.9489, 0.872, 0.9686, 0.9894] 
+2024-09-06 18:59:07.664695: Epoch time: 346.29 s 
+2024-09-06 18:59:10.125341:  
+2024-09-06 18:59:10.126042: Epoch 415 
+2024-09-06 18:59:10.127111: Current learning rate: 0.00617 
+2024-09-06 19:05:56.768124: train_loss -0.7397 
+2024-09-06 19:05:56.772161: val_loss -0.7365 
+2024-09-06 19:05:56.773006: Pseudo dice [0.9637, 0.9561, 0.9788, 0.9435, 0.8269, 0.9555, 0.9807] 
+2024-09-06 19:05:56.774385: Epoch time: 406.65 s 
+2024-09-06 19:06:02.864478:  
+2024-09-06 19:06:02.865355: Epoch 416 
+2024-09-06 19:06:02.866163: Current learning rate: 0.00616 
+2024-09-06 19:12:48.628213: train_loss -0.7281 
+2024-09-06 19:12:48.645880: val_loss -0.7495 
+2024-09-06 19:12:48.646564: Pseudo dice [0.9611, 0.9531, 0.9761, 0.9418, 0.8325, 0.9608, 0.9872] 
+2024-09-06 19:12:48.647083: Epoch time: 405.86 s 
+2024-09-06 19:12:51.026134:  
+2024-09-06 19:12:51.027147: Epoch 417 
+2024-09-06 19:12:51.027911: Current learning rate: 0.00615 
+2024-09-06 19:18:44.657540: train_loss -0.7391 
+2024-09-06 19:18:44.671350: val_loss -0.7399 
+2024-09-06 19:18:44.672131: Pseudo dice [0.9613, 0.9565, 0.9773, 0.9507, 0.8829, 0.9672, 0.9904] 
+2024-09-06 19:18:44.672861: Epoch time: 353.64 s 
+2024-09-06 19:18:48.559953:  
+2024-09-06 19:18:48.560610: Epoch 418 
+2024-09-06 19:18:48.561154: Current learning rate: 0.00614 
+2024-09-06 19:24:21.152240: train_loss -0.7202 
+2024-09-06 19:24:21.154925: val_loss -0.7158 
+2024-09-06 19:24:21.155585: Pseudo dice [0.9595, 0.9534, 0.9734, 0.944, 0.8572, 0.9657, 0.9892] 
+2024-09-06 19:24:21.156106: Epoch time: 332.6 s 
+2024-09-06 19:24:24.336041:  
+2024-09-06 19:24:24.336736: Epoch 419 
+2024-09-06 19:24:24.337244: Current learning rate: 0.00613 
+2024-09-06 19:30:26.067106: train_loss -0.7264 
+2024-09-06 19:30:26.081312: val_loss -0.6977 
+2024-09-06 19:30:26.082131: Pseudo dice [0.9632, 0.9547, 0.9784, 0.948, 0.8727, 0.9676, 0.9916] 
+2024-09-06 19:30:26.082918: Epoch time: 361.73 s 
+2024-09-06 19:30:29.185604:  
+2024-09-06 19:30:29.186596: Epoch 420 
+2024-09-06 19:30:29.187252: Current learning rate: 0.00612 
+2024-09-06 19:35:55.590300: train_loss -0.7241 
+2024-09-06 19:35:55.604800: val_loss -0.6999 
+2024-09-06 19:35:55.605349: Pseudo dice [0.9625, 0.9544, 0.9764, 0.9472, 0.8456, 0.9651, 0.9877] 
+2024-09-06 19:35:55.605827: Epoch time: 326.41 s 
+2024-09-06 19:35:58.152256:  
+2024-09-06 19:35:58.153470: Epoch 421 
+2024-09-06 19:35:58.154444: Current learning rate: 0.00612 
+2024-09-06 19:41:31.942719: train_loss -0.709 
+2024-09-06 19:41:31.966460: val_loss -0.7129 
+2024-09-06 19:41:31.967280: Pseudo dice [0.957, 0.9508, 0.9753, 0.941, 0.8602, 0.9662, 0.9886] 
+2024-09-06 19:41:31.967947: Epoch time: 333.8 s 
+2024-09-06 19:41:37.166809:  
+2024-09-06 19:41:37.180453: Epoch 422 
+2024-09-06 19:41:37.181297: Current learning rate: 0.00611 
+2024-09-06 19:47:37.667884: train_loss -0.7151 
+2024-09-06 19:47:37.669918: val_loss -0.7269 
+2024-09-06 19:47:37.670572: Pseudo dice [0.9588, 0.9553, 0.973, 0.9413, 0.8482, 0.9649, 0.9883] 
+2024-09-06 19:47:37.671103: Epoch time: 360.51 s 
+2024-09-06 19:47:39.653139:  
+2024-09-06 19:47:39.653755: Epoch 423 
+2024-09-06 19:47:39.654217: Current learning rate: 0.0061 
+2024-09-06 19:52:32.861964: train_loss -0.7231 
+2024-09-06 19:52:32.878186: val_loss -0.7224 
+2024-09-06 19:52:32.879109: Pseudo dice [0.9611, 0.9565, 0.9773, 0.9434, 0.8753, 0.9683, 0.9907] 
+2024-09-06 19:52:32.879884: Epoch time: 293.21 s 
+2024-09-06 19:52:35.909427:  
+2024-09-06 19:52:35.910486: Epoch 424 
+2024-09-06 19:52:35.912039: Current learning rate: 0.00609 
+2024-09-06 19:57:21.784041: train_loss -0.7257 
+2024-09-06 19:57:21.801421: val_loss -0.7617 
+2024-09-06 19:57:21.801980: Pseudo dice [0.9629, 0.9555, 0.9785, 0.9483, 0.8783, 0.9633, 0.9906] 
+2024-09-06 19:57:21.802502: Epoch time: 285.88 s 
+2024-09-06 19:57:24.191899:  
+2024-09-06 19:57:24.208611: Epoch 425 
+2024-09-06 19:57:24.209673: Current learning rate: 0.00608 
+2024-09-06 20:02:29.516250: train_loss -0.7186 
+2024-09-06 20:02:29.529777: val_loss -0.7231 
+2024-09-06 20:02:29.530351: Pseudo dice [0.9552, 0.9543, 0.9667, 0.9435, 0.8716, 0.9668, 0.9899] 
+2024-09-06 20:02:29.530788: Epoch time: 305.33 s 
+2024-09-06 20:02:32.633964:  
+2024-09-06 20:02:32.635334: Epoch 426 
+2024-09-06 20:02:32.636091: Current learning rate: 0.00607 
+2024-09-06 20:07:33.497641: train_loss -0.7391 
+2024-09-06 20:07:33.513557: val_loss -0.7144 
+2024-09-06 20:07:33.514077: Pseudo dice [0.9618, 0.9536, 0.9741, 0.9494, 0.8562, 0.9647, 0.9893] 
+2024-09-06 20:07:33.514690: Epoch time: 300.87 s 
+2024-09-06 20:07:36.466695:  
+2024-09-06 20:07:36.479996: Epoch 427 
+2024-09-06 20:07:36.480618: Current learning rate: 0.00606 
+2024-09-06 20:12:12.475966: train_loss -0.7305 
+2024-09-06 20:12:12.493407: val_loss -0.7203 
+2024-09-06 20:12:12.494081: Pseudo dice [0.9592, 0.9557, 0.9723, 0.947, 0.877, 0.9574, 0.9872] 
+2024-09-06 20:12:12.494557: Epoch time: 276.01 s 
+2024-09-06 20:12:15.952065:  
+2024-09-06 20:12:15.967752: Epoch 428 
+2024-09-06 20:12:15.968550: Current learning rate: 0.00605 
+2024-09-06 20:16:47.871441: train_loss -0.7397 
+2024-09-06 20:16:47.893573: val_loss -0.7362 
+2024-09-06 20:16:47.894141: Pseudo dice [0.9622, 0.9552, 0.9791, 0.9481, 0.8316, 0.9714, 0.9861] 
+2024-09-06 20:16:47.894645: Epoch time: 271.92 s 
+2024-09-06 20:16:50.800658:  
+2024-09-06 20:16:50.806758: Epoch 429 
+2024-09-06 20:16:50.808310: Current learning rate: 0.00604 
+2024-09-06 20:21:21.571056: train_loss -0.7312 
+2024-09-06 20:21:21.573888: val_loss -0.7488 
+2024-09-06 20:21:21.574526: Pseudo dice [0.963, 0.9576, 0.9788, 0.9448, 0.876, 0.97, 0.9909] 
+2024-09-06 20:21:21.575206: Epoch time: 270.77 s 
+2024-09-06 20:21:24.073954:  
+2024-09-06 20:21:24.074999: Epoch 430 
+2024-09-06 20:21:24.075841: Current learning rate: 0.00603 
+2024-09-06 20:25:49.865250: train_loss -0.7209 
+2024-09-06 20:25:49.867080: val_loss -0.7246 
+2024-09-06 20:25:49.867563: Pseudo dice [0.9652, 0.9575, 0.9801, 0.9485, 0.8673, 0.969, 0.9902] 
+2024-09-06 20:25:49.868127: Epoch time: 265.8 s 
+2024-09-06 20:25:52.644730:  
+2024-09-06 20:25:52.645442: Epoch 431 
+2024-09-06 20:25:52.645979: Current learning rate: 0.00602 
+2024-09-06 20:29:59.282262: train_loss -0.7405 
+2024-09-06 20:29:59.298391: val_loss -0.7605 
+2024-09-06 20:29:59.299408: Pseudo dice [0.9633, 0.9553, 0.978, 0.9482, 0.84, 0.9518, 0.9806] 
+2024-09-06 20:29:59.300223: Epoch time: 246.64 s 
+2024-09-06 20:30:01.232596:  
+2024-09-06 20:30:01.243752: Epoch 432 
+2024-09-06 20:30:01.244549: Current learning rate: 0.00601 
+2024-09-06 20:34:31.200018: train_loss -0.7177 
+2024-09-06 20:34:31.202907: val_loss -0.7446 
+2024-09-06 20:34:31.203540: Pseudo dice [0.9648, 0.9582, 0.9799, 0.95, 0.8648, 0.9686, 0.99] 
+2024-09-06 20:34:31.204087: Epoch time: 269.97 s 
+2024-09-06 20:34:35.054965:  
+2024-09-06 20:34:35.055710: Epoch 433 
+2024-09-06 20:34:35.056215: Current learning rate: 0.006 
+2024-09-06 20:39:00.752800: train_loss -0.7241 
+2024-09-06 20:39:00.763984: val_loss -0.7242 
+2024-09-06 20:39:00.764506: Pseudo dice [0.9603, 0.9591, 0.9737, 0.9474, 0.8641, 0.9572, 0.9853] 
+2024-09-06 20:39:00.765291: Epoch time: 265.7 s 
+2024-09-06 20:39:02.983848:  
+2024-09-06 20:39:02.985054: Epoch 434 
+2024-09-06 20:39:02.986631: Current learning rate: 0.00599 
+2024-09-06 20:43:15.911738: train_loss -0.7236 
+2024-09-06 20:43:15.913710: val_loss -0.7325 
+2024-09-06 20:43:15.914171: Pseudo dice [0.9616, 0.953, 0.9769, 0.9378, 0.8495, 0.9505, 0.9863] 
+2024-09-06 20:43:15.914611: Epoch time: 252.93 s 
+2024-09-06 20:43:18.225527:  
+2024-09-06 20:43:18.226233: Epoch 435 
+2024-09-06 20:43:18.226729: Current learning rate: 0.00598 
+2024-09-06 20:47:38.560026: train_loss -0.7212 
+2024-09-06 20:47:38.562544: val_loss -0.7314 
+2024-09-06 20:47:38.563092: Pseudo dice [0.9634, 0.957, 0.9774, 0.9427, 0.8546, 0.9603, 0.9864] 
+2024-09-06 20:47:38.563604: Epoch time: 260.34 s 
+2024-09-06 20:47:41.508625:  
+2024-09-06 20:47:41.509687: Epoch 436 
+2024-09-06 20:47:41.511312: Current learning rate: 0.00597 
+2024-09-06 20:52:04.036726: train_loss -0.745 
+2024-09-06 20:52:04.053630: val_loss -0.7211 
+2024-09-06 20:52:04.054122: Pseudo dice [0.9601, 0.9567, 0.9741, 0.9459, 0.845, 0.9546, 0.9809] 
+2024-09-06 20:52:04.054652: Epoch time: 262.53 s 
+2024-09-06 20:52:06.713593:  
+2024-09-06 20:52:06.721968: Epoch 437 
+2024-09-06 20:52:06.722636: Current learning rate: 0.00596 
+2024-09-06 20:56:28.283972: train_loss -0.7264 
+2024-09-06 20:56:28.309575: val_loss -0.7373 
+2024-09-06 20:56:28.310080: Pseudo dice [0.9629, 0.9561, 0.9769, 0.9479, 0.8796, 0.9703, 0.9906] 
+2024-09-06 20:56:28.310509: Epoch time: 261.57 s 
+2024-09-06 20:56:31.848563:  
+2024-09-06 20:56:31.866300: Epoch 438 
+2024-09-06 20:56:31.867204: Current learning rate: 0.00595 
+2024-09-06 21:00:57.411681: train_loss -0.733 
+2024-09-06 21:00:57.425348: val_loss -0.7297 
+2024-09-06 21:00:57.425982: Pseudo dice [0.9651, 0.9553, 0.9779, 0.9441, 0.8566, 0.9597, 0.9871] 
+2024-09-06 21:00:57.426462: Epoch time: 265.59 s 
+2024-09-06 21:01:00.469901:  
+2024-09-06 21:01:00.470716: Epoch 439 
+2024-09-06 21:01:00.471376: Current learning rate: 0.00594 
+2024-09-06 21:05:38.748994: train_loss -0.7275 
+2024-09-06 21:05:38.765310: val_loss -0.7755 
+2024-09-06 21:05:38.765841: Pseudo dice [0.9634, 0.9571, 0.9797, 0.9475, 0.8625, 0.9647, 0.9901] 
+2024-09-06 21:05:38.766284: Epoch time: 278.28 s 
+2024-09-06 21:05:41.837699:  
+2024-09-06 21:05:41.838524: Epoch 440 
+2024-09-06 21:05:41.839131: Current learning rate: 0.00593 
+2024-09-06 21:10:02.078124: train_loss -0.7335 
+2024-09-06 21:10:02.080365: val_loss -0.7334 
+2024-09-06 21:10:02.080945: Pseudo dice [0.9627, 0.9566, 0.9774, 0.9452, 0.867, 0.9659, 0.9873] 
+2024-09-06 21:10:02.081464: Epoch time: 260.25 s 
+2024-09-06 21:10:04.875108:  
+2024-09-06 21:10:04.888159: Epoch 441 
+2024-09-06 21:10:04.888772: Current learning rate: 0.00592 
+2024-09-06 21:14:18.617335: train_loss -0.7388 
+2024-09-06 21:14:18.644519: val_loss -0.7316 
+2024-09-06 21:14:18.645116: Pseudo dice [0.9657, 0.9584, 0.9806, 0.9508, 0.8771, 0.9688, 0.9913] 
+2024-09-06 21:14:18.645566: Epoch time: 253.75 s 
+2024-09-06 21:14:20.944518:  
+2024-09-06 21:14:20.960045: Epoch 442 
+2024-09-06 21:14:20.960897: Current learning rate: 0.00592 
+2024-09-06 21:19:05.521219: train_loss -0.7484 
+2024-09-06 21:19:05.536276: val_loss -0.7439 
+2024-09-06 21:19:05.536901: Pseudo dice [0.965, 0.9573, 0.978, 0.9464, 0.8563, 0.9664, 0.9906] 
+2024-09-06 21:19:05.537727: Epoch time: 284.58 s 
+2024-09-06 21:19:09.137648:  
+2024-09-06 21:19:09.156127: Epoch 443 
+2024-09-06 21:19:09.157147: Current learning rate: 0.00591 
+2024-09-06 21:23:17.328776: train_loss -0.7343 
+2024-09-06 21:23:17.336023: val_loss -0.7625 
+2024-09-06 21:23:17.336533: Pseudo dice [0.9627, 0.9587, 0.9765, 0.9497, 0.8717, 0.9696, 0.9908] 
+2024-09-06 21:23:17.336994: Epoch time: 248.2 s 
+2024-09-06 21:23:20.503945:  
+2024-09-06 21:23:20.504543: Epoch 444 
+2024-09-06 21:23:20.505008: Current learning rate: 0.0059 
+2024-09-06 21:27:50.002724: train_loss -0.7415 
+2024-09-06 21:27:50.016312: val_loss -0.7206 
+2024-09-06 21:27:50.016997: Pseudo dice [0.9641, 0.9582, 0.977, 0.9457, 0.8504, 0.9438, 0.9835] 
+2024-09-06 21:27:50.017582: Epoch time: 269.5 s 
+2024-09-06 21:27:52.916221:  
+2024-09-06 21:27:52.924209: Epoch 445 
+2024-09-06 21:27:52.925005: Current learning rate: 0.00589 
+2024-09-06 21:32:34.263280: train_loss -0.7358 
+2024-09-06 21:32:34.289190: val_loss -0.7583 
+2024-09-06 21:32:34.289742: Pseudo dice [0.9632, 0.9565, 0.9763, 0.9469, 0.8778, 0.969, 0.991] 
+2024-09-06 21:32:34.290203: Epoch time: 281.35 s 
+2024-09-06 21:32:37.176885:  
+2024-09-06 21:32:37.177518: Epoch 446 
+2024-09-06 21:32:37.178272: Current learning rate: 0.00588 
+2024-09-06 21:37:35.828228: train_loss -0.7391 
+2024-09-06 21:37:35.840137: val_loss -0.7045 
+2024-09-06 21:37:35.840896: Pseudo dice [0.9605, 0.9513, 0.9759, 0.9492, 0.857, 0.9528, 0.9872] 
+2024-09-06 21:37:35.841449: Epoch time: 298.66 s 
+2024-09-06 21:37:40.377759:  
+2024-09-06 21:37:40.378397: Epoch 447 
+2024-09-06 21:37:40.378823: Current learning rate: 0.00587 
+2024-09-06 21:42:29.529096: train_loss -0.7425 
+2024-09-06 21:42:29.541913: val_loss -0.7334 
+2024-09-06 21:42:29.544212: Pseudo dice [0.9611, 0.9555, 0.9729, 0.9387, 0.8654, 0.965, 0.9898] 
+2024-09-06 21:42:29.544709: Epoch time: 289.16 s 
+2024-09-06 21:42:34.402220:  
+2024-09-06 21:42:34.416421: Epoch 448 
+2024-09-06 21:42:34.417199: Current learning rate: 0.00586 
+2024-09-06 21:47:18.146460: train_loss -0.7278 
+2024-09-06 21:47:18.159294: val_loss -0.7279 
+2024-09-06 21:47:18.160074: Pseudo dice [0.9647, 0.958, 0.9812, 0.9496, 0.8616, 0.9666, 0.9898] 
+2024-09-06 21:47:18.161088: Epoch time: 283.75 s 
+2024-09-06 21:47:21.403991:  
+2024-09-06 21:47:21.413511: Epoch 449 
+2024-09-06 21:47:21.414217: Current learning rate: 0.00585 
+2024-09-06 21:51:59.711279: train_loss -0.7179 
+2024-09-06 21:51:59.725250: val_loss -0.7098 
+2024-09-06 21:51:59.725861: Pseudo dice [0.9642, 0.9556, 0.9776, 0.9442, 0.8428, 0.9689, 0.9849] 
+2024-09-06 21:51:59.726341: Epoch time: 278.31 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_7_14_05_16.txt

@@ -0,0 +1,373 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-07 14:05:19.560474: Using torch.compile... 
+2024-09-07 14:05:25.981516: do_dummy_2d_data_aug: False 
+2024-09-07 14:05:25.987476: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-07 14:05:25.989626: The split file contains 5 splits. 
+2024-09-07 14:05:25.990552: Desired fold for training: 1 
+2024-09-07 14:05:25.991089: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-07 14:05:43.841383: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-07 14:05:45.582211:  
+2024-09-07 14:05:45.600064: Epoch 450 
+2024-09-07 14:05:45.604346: Current learning rate: 0.00584 
+2024-09-07 14:24:14.053076: train_loss -0.7444 
+2024-09-07 14:24:14.055376: val_loss -0.7604 
+2024-09-07 14:24:14.056417: Pseudo dice [0.9649, 0.9565, 0.9781, 0.95, 0.874, 0.956, 0.9896] 
+2024-09-07 14:24:14.057210: Epoch time: 1108.48 s 
+2024-09-07 14:24:16.646980:  
+2024-09-07 14:24:16.648176: Epoch 451 
+2024-09-07 14:24:16.648967: Current learning rate: 0.00583 
+2024-09-07 14:31:50.110551: train_loss -0.74 
+2024-09-07 14:31:50.129311: val_loss -0.7347 
+2024-09-07 14:31:50.130499: Pseudo dice [0.9633, 0.958, 0.9788, 0.9483, 0.8462, 0.9688, 0.9887] 
+2024-09-07 14:31:50.131449: Epoch time: 453.47 s 
+2024-09-07 14:31:53.875885:  
+2024-09-07 14:31:53.877291: Epoch 452 
+2024-09-07 14:31:53.878617: Current learning rate: 0.00582 
+2024-09-07 14:39:26.670203: train_loss -0.7454 
+2024-09-07 14:39:26.673620: val_loss -0.7458 
+2024-09-07 14:39:26.675415: Pseudo dice [0.9651, 0.9542, 0.9803, 0.949, 0.8564, 0.9583, 0.9855] 
+2024-09-07 14:39:26.676216: Epoch time: 452.8 s 
+2024-09-07 14:39:30.711248:  
+2024-09-07 14:39:30.727787: Epoch 453 
+2024-09-07 14:39:30.729001: Current learning rate: 0.00581 
+2024-09-07 14:47:29.440723: train_loss -0.7145 
+2024-09-07 14:47:29.448656: val_loss -0.7343 
+2024-09-07 14:47:29.450079: Pseudo dice [0.961, 0.9557, 0.9737, 0.9446, 0.8591, 0.9618, 0.988] 
+2024-09-07 14:47:29.451703: Epoch time: 478.74 s 
+2024-09-07 14:47:33.321514:  
+2024-09-07 14:47:33.322760: Epoch 454 
+2024-09-07 14:47:33.323783: Current learning rate: 0.0058 
+2024-09-07 14:55:49.645158: train_loss -0.7346 
+2024-09-07 14:55:49.664622: val_loss -0.7009 
+2024-09-07 14:55:49.665489: Pseudo dice [0.9614, 0.9552, 0.9742, 0.9446, 0.8649, 0.9628, 0.9897] 
+2024-09-07 14:55:49.666220: Epoch time: 496.34 s 
+2024-09-07 14:55:56.431030:  
+2024-09-07 14:55:56.432599: Epoch 455 
+2024-09-07 14:55:56.433487: Current learning rate: 0.00579 
+2024-09-07 15:04:10.354673: train_loss -0.7255 
+2024-09-07 15:04:10.376522: val_loss -0.7568 
+2024-09-07 15:04:10.387614: Pseudo dice [0.9632, 0.9565, 0.9782, 0.9451, 0.8717, 0.9569, 0.9885] 
+2024-09-07 15:04:10.392314: Epoch time: 493.93 s 
+2024-09-07 15:04:14.683731:  
+2024-09-07 15:04:14.699590: Epoch 456 
+2024-09-07 15:04:14.703272: Current learning rate: 0.00578 
+2024-09-07 15:11:15.463906: train_loss -0.7526 
+2024-09-07 15:11:15.474091: val_loss -0.6902 
+2024-09-07 15:11:15.475332: Pseudo dice [0.9634, 0.957, 0.978, 0.9486, 0.8469, 0.9551, 0.9846] 
+2024-09-07 15:11:15.476855: Epoch time: 420.79 s 
+2024-09-07 15:11:20.276490:  
+2024-09-07 15:11:20.292882: Epoch 457 
+2024-09-07 15:11:20.293828: Current learning rate: 0.00577 
+2024-09-07 15:18:29.490092: train_loss -0.7342 
+2024-09-07 15:18:29.519701: val_loss -0.7775 
+2024-09-07 15:18:29.521712: Pseudo dice [0.9657, 0.9567, 0.9811, 0.9555, 0.8791, 0.9694, 0.993] 
+2024-09-07 15:18:29.523133: Epoch time: 429.22 s 
+2024-09-07 15:18:36.295188:  
+2024-09-07 15:18:36.298867: Epoch 458 
+2024-09-07 15:18:36.302421: Current learning rate: 0.00576 
+2024-09-07 15:25:52.996993: train_loss -0.7424 
+2024-09-07 15:25:53.018279: val_loss -0.7462 
+2024-09-07 15:25:53.021342: Pseudo dice [0.9623, 0.9584, 0.9743, 0.9443, 0.8634, 0.9677, 0.9877] 
+2024-09-07 15:25:53.022325: Epoch time: 436.73 s 
+2024-09-07 15:25:57.567185:  
+2024-09-07 15:25:57.580750: Epoch 459 
+2024-09-07 15:25:57.582550: Current learning rate: 0.00575 
+2024-09-07 15:33:22.499581: train_loss -0.7472 
+2024-09-07 15:33:22.513659: val_loss -0.718 
+2024-09-07 15:33:22.514517: Pseudo dice [0.9604, 0.9547, 0.9757, 0.9419, 0.8635, 0.9625, 0.9903] 
+2024-09-07 15:33:22.515227: Epoch time: 444.94 s 
+2024-09-07 15:33:26.403259:  
+2024-09-07 15:33:26.404909: Epoch 460 
+2024-09-07 15:33:26.405802: Current learning rate: 0.00574 
+2024-09-07 15:41:03.716638: train_loss -0.7302 
+2024-09-07 15:41:03.719032: val_loss -0.747 
+2024-09-07 15:41:03.719777: Pseudo dice [0.9629, 0.9555, 0.9783, 0.9471, 0.855, 0.967, 0.9894] 
+2024-09-07 15:41:03.720469: Epoch time: 457.32 s 
+2024-09-07 15:41:07.473203:  
+2024-09-07 15:41:07.474758: Epoch 461 
+2024-09-07 15:41:07.476378: Current learning rate: 0.00573 
+2024-09-07 15:48:16.466712: train_loss -0.7195 
+2024-09-07 15:48:16.476370: val_loss -0.7323 
+2024-09-07 15:48:16.478272: Pseudo dice [0.9631, 0.9577, 0.9786, 0.9441, 0.868, 0.9616, 0.9876] 
+2024-09-07 15:48:16.480057: Epoch time: 429.0 s 
+2024-09-07 15:48:20.258442:  
+2024-09-07 15:48:20.271150: Epoch 462 
+2024-09-07 15:48:20.272856: Current learning rate: 0.00572 
+2024-09-07 15:55:18.583315: train_loss -0.7187 
+2024-09-07 15:55:18.587054: val_loss -0.7234 
+2024-09-07 15:55:18.589132: Pseudo dice [0.9654, 0.9569, 0.9798, 0.9458, 0.8645, 0.9473, 0.9848] 
+2024-09-07 15:55:18.591212: Epoch time: 418.36 s 
+2024-09-07 15:55:23.100567:  
+2024-09-07 15:55:23.101794: Epoch 463 
+2024-09-07 15:55:23.104567: Current learning rate: 0.00571 
+2024-09-07 16:02:22.950456: train_loss -0.732 
+2024-09-07 16:02:22.975701: val_loss -0.7223 
+2024-09-07 16:02:22.976630: Pseudo dice [0.9627, 0.9546, 0.9757, 0.948, 0.8616, 0.9591, 0.9888] 
+2024-09-07 16:02:22.977304: Epoch time: 419.87 s 
+2024-09-07 16:02:27.184300:  
+2024-09-07 16:02:27.186159: Epoch 464 
+2024-09-07 16:02:27.187334: Current learning rate: 0.0057 
+2024-09-07 16:09:33.288958: train_loss -0.7485 
+2024-09-07 16:09:33.305279: val_loss -0.7474 
+2024-09-07 16:09:33.306281: Pseudo dice [0.9663, 0.9572, 0.9801, 0.9442, 0.8738, 0.9659, 0.9916] 
+2024-09-07 16:09:33.307178: Epoch time: 426.14 s 
+2024-09-07 16:09:36.952105:  
+2024-09-07 16:09:36.967840: Epoch 465 
+2024-09-07 16:09:36.974753: Current learning rate: 0.0057 
+2024-09-07 16:16:29.701430: train_loss -0.7329 
+2024-09-07 16:16:29.706266: val_loss -0.7623 
+2024-09-07 16:16:29.707710: Pseudo dice [0.959, 0.9535, 0.9723, 0.9497, 0.8766, 0.956, 0.9896] 
+2024-09-07 16:16:29.709680: Epoch time: 412.76 s 
+2024-09-07 16:16:41.358479:  
+2024-09-07 16:16:41.370647: Epoch 466 
+2024-09-07 16:16:41.372231: Current learning rate: 0.00569 
+2024-09-07 16:24:04.283282: train_loss -0.7175 
+2024-09-07 16:24:04.300947: val_loss -0.7336 
+2024-09-07 16:24:04.302794: Pseudo dice [0.9611, 0.9553, 0.9766, 0.9409, 0.8572, 0.9503, 0.9866] 
+2024-09-07 16:24:04.304275: Epoch time: 442.93 s 
+2024-09-07 16:24:07.649179:  
+2024-09-07 16:24:07.651943: Epoch 467 
+2024-09-07 16:24:07.652869: Current learning rate: 0.00568 
+2024-09-07 16:31:01.005773: train_loss -0.7355 
+2024-09-07 16:31:01.029804: val_loss -0.7287 
+2024-09-07 16:31:01.031934: Pseudo dice [0.9613, 0.9545, 0.9757, 0.9446, 0.866, 0.9515, 0.9881] 
+2024-09-07 16:31:01.033741: Epoch time: 413.36 s 
+2024-09-07 16:31:05.633139:  
+2024-09-07 16:31:05.650280: Epoch 468 
+2024-09-07 16:31:05.651988: Current learning rate: 0.00567 
+2024-09-07 16:38:10.382010: train_loss -0.7213 
+2024-09-07 16:38:10.397908: val_loss -0.6952 
+2024-09-07 16:38:10.399674: Pseudo dice [0.9588, 0.9519, 0.9715, 0.9329, 0.8339, 0.9495, 0.9834] 
+2024-09-07 16:38:10.402426: Epoch time: 424.75 s 
+2024-09-07 16:38:13.957599:  
+2024-09-07 16:38:13.973252: Epoch 469 
+2024-09-07 16:38:13.974936: Current learning rate: 0.00566 
+2024-09-07 16:45:32.028459: train_loss -0.6965 
+2024-09-07 16:45:32.037712: val_loss -0.7355 
+2024-09-07 16:45:32.038717: Pseudo dice [0.9592, 0.9487, 0.9752, 0.9356, 0.8591, 0.9665, 0.9908] 
+2024-09-07 16:45:32.039531: Epoch time: 438.08 s 
+2024-09-07 16:45:37.020987:  
+2024-09-07 16:45:37.051894: Epoch 470 
+2024-09-07 16:45:37.055070: Current learning rate: 0.00565 
+2024-09-07 16:53:00.403705: train_loss -0.7024 
+2024-09-07 16:53:00.407124: val_loss -0.7068 
+2024-09-07 16:53:00.408577: Pseudo dice [0.9603, 0.9501, 0.9724, 0.9432, 0.8277, 0.9656, 0.9877] 
+2024-09-07 16:53:00.410167: Epoch time: 443.39 s 
+2024-09-07 16:53:06.428147:  
+2024-09-07 16:53:06.443900: Epoch 471 
+2024-09-07 16:53:06.446186: Current learning rate: 0.00564 
+2024-09-07 16:59:57.550220: train_loss -0.7272 
+2024-09-07 16:59:57.569263: val_loss -0.7309 
+2024-09-07 16:59:57.570448: Pseudo dice [0.96, 0.9557, 0.9724, 0.948, 0.8675, 0.9674, 0.9913] 
+2024-09-07 16:59:57.571888: Epoch time: 411.13 s 
+2024-09-07 17:00:01.683234:  
+2024-09-07 17:00:01.697786: Epoch 472 
+2024-09-07 17:00:01.699291: Current learning rate: 0.00563 
+2024-09-07 17:07:17.128064: train_loss -0.7339 
+2024-09-07 17:07:17.131284: val_loss -0.7364 
+2024-09-07 17:07:17.133998: Pseudo dice [0.9623, 0.9564, 0.9773, 0.9471, 0.8755, 0.9559, 0.9883] 
+2024-09-07 17:07:17.136229: Epoch time: 435.46 s 
+2024-09-07 17:07:21.719164:  
+2024-09-07 17:07:21.728770: Epoch 473 
+2024-09-07 17:07:21.730186: Current learning rate: 0.00562 
+2024-09-07 17:14:33.611333: train_loss -0.7337 
+2024-09-07 17:14:33.615066: val_loss -0.7386 
+2024-09-07 17:14:33.617495: Pseudo dice [0.9571, 0.9549, 0.9704, 0.9373, 0.8656, 0.9672, 0.9906] 
+2024-09-07 17:14:33.619527: Epoch time: 431.9 s 
+2024-09-07 17:14:37.831811:  
+2024-09-07 17:14:37.845005: Epoch 474 
+2024-09-07 17:14:37.846279: Current learning rate: 0.00561 
+2024-09-07 17:21:42.755908: train_loss -0.7369 
+2024-09-07 17:21:42.765899: val_loss -0.7355 
+2024-09-07 17:21:42.767559: Pseudo dice [0.9629, 0.9567, 0.9794, 0.9517, 0.8772, 0.958, 0.9874] 
+2024-09-07 17:21:42.768526: Epoch time: 424.93 s 
+2024-09-07 17:21:45.991551:  
+2024-09-07 17:21:45.996121: Epoch 475 
+2024-09-07 17:21:45.998345: Current learning rate: 0.0056 
+2024-09-07 17:28:33.843607: train_loss -0.7226 
+2024-09-07 17:28:33.848171: val_loss -0.7342 
+2024-09-07 17:28:33.850610: Pseudo dice [0.9541, 0.9516, 0.9685, 0.9369, 0.8513, 0.9662, 0.986] 
+2024-09-07 17:28:33.852985: Epoch time: 407.88 s 
+2024-09-07 17:28:39.615159:  
+2024-09-07 17:28:39.633088: Epoch 476 
+2024-09-07 17:28:39.636145: Current learning rate: 0.00559 
+2024-09-07 17:35:20.825737: train_loss -0.7364 
+2024-09-07 17:35:20.828707: val_loss -0.7476 
+2024-09-07 17:35:20.830656: Pseudo dice [0.9596, 0.9583, 0.9745, 0.9554, 0.876, 0.9699, 0.9898] 
+2024-09-07 17:35:20.831704: Epoch time: 401.24 s 
+2024-09-07 17:35:25.009374:  
+2024-09-07 17:35:25.016883: Epoch 477 
+2024-09-07 17:35:25.017999: Current learning rate: 0.00558 
+2024-09-07 17:42:21.825239: train_loss -0.7352 
+2024-09-07 17:42:21.850258: val_loss -0.7191 
+2024-09-07 17:42:21.851419: Pseudo dice [0.9632, 0.9577, 0.9776, 0.94, 0.8332, 0.9649, 0.9871] 
+2024-09-07 17:42:21.852825: Epoch time: 416.82 s 
+2024-09-07 17:42:27.363130:  
+2024-09-07 17:42:27.372869: Epoch 478 
+2024-09-07 17:42:27.375530: Current learning rate: 0.00557 
+2024-09-07 17:49:34.520597: train_loss -0.735 
+2024-09-07 17:49:34.547544: val_loss -0.7564 
+2024-09-07 17:49:34.548667: Pseudo dice [0.9669, 0.9581, 0.9793, 0.9478, 0.8763, 0.9657, 0.9915] 
+2024-09-07 17:49:34.550246: Epoch time: 427.19 s 
+2024-09-07 17:49:39.117971:  
+2024-09-07 17:49:39.124814: Epoch 479 
+2024-09-07 17:49:39.125789: Current learning rate: 0.00556 
+2024-09-07 17:56:30.838347: train_loss -0.7466 
+2024-09-07 17:56:30.842175: val_loss -0.7273 
+2024-09-07 17:56:30.843230: Pseudo dice [0.9614, 0.9525, 0.9703, 0.9402, 0.8398, 0.9674, 0.9865] 
+2024-09-07 17:56:30.844037: Epoch time: 411.73 s 
+2024-09-07 17:56:36.078005:  
+2024-09-07 17:56:36.084642: Epoch 480 
+2024-09-07 17:56:36.086105: Current learning rate: 0.00555 
+2024-09-07 18:03:35.773607: train_loss -0.7354 
+2024-09-07 18:03:35.794289: val_loss -0.7646 
+2024-09-07 18:03:35.796064: Pseudo dice [0.9625, 0.9575, 0.9788, 0.9454, 0.8769, 0.968, 0.9925] 
+2024-09-07 18:03:35.798887: Epoch time: 419.71 s 
+2024-09-07 18:03:45.129850:  
+2024-09-07 18:03:45.144995: Epoch 481 
+2024-09-07 18:03:45.146458: Current learning rate: 0.00554 
+2024-09-07 18:10:34.869570: train_loss -0.7283 
+2024-09-07 18:10:34.885940: val_loss -0.7382 
+2024-09-07 18:10:34.887567: Pseudo dice [0.9641, 0.9583, 0.9777, 0.9452, 0.8573, 0.9663, 0.9855] 
+2024-09-07 18:10:34.888602: Epoch time: 409.75 s 
+2024-09-07 18:10:38.943146:  
+2024-09-07 18:10:38.956925: Epoch 482 
+2024-09-07 18:10:38.957881: Current learning rate: 0.00553 
+2024-09-07 18:18:14.417495: train_loss -0.7375 
+2024-09-07 18:18:14.433619: val_loss -0.7311 
+2024-09-07 18:18:14.434977: Pseudo dice [0.9607, 0.9566, 0.9752, 0.9454, 0.851, 0.9572, 0.9853] 
+2024-09-07 18:18:14.436162: Epoch time: 455.48 s 
+2024-09-07 18:18:18.340506:  
+2024-09-07 18:18:18.353173: Epoch 483 
+2024-09-07 18:18:18.354883: Current learning rate: 0.00552 
+2024-09-07 18:25:12.150919: train_loss -0.7347 
+2024-09-07 18:25:12.173210: val_loss -0.7365 
+2024-09-07 18:25:12.175915: Pseudo dice [0.9647, 0.9574, 0.9794, 0.9501, 0.8628, 0.9652, 0.9882] 
+2024-09-07 18:25:12.179572: Epoch time: 413.86 s 
+2024-09-07 18:25:16.567327:  
+2024-09-07 18:25:16.587813: Epoch 484 
+2024-09-07 18:25:16.589131: Current learning rate: 0.00551 
+2024-09-07 18:32:04.114299: train_loss -0.754 
+2024-09-07 18:32:04.137744: val_loss -0.7647 
+2024-09-07 18:32:04.139642: Pseudo dice [0.9642, 0.9561, 0.9779, 0.9453, 0.8773, 0.9696, 0.9911] 
+2024-09-07 18:32:04.141192: Epoch time: 407.55 s 
+2024-09-07 18:32:08.704132:  
+2024-09-07 18:32:08.716792: Epoch 485 
+2024-09-07 18:32:08.717748: Current learning rate: 0.0055 
+2024-09-07 18:38:55.549684: train_loss -0.7379 
+2024-09-07 18:38:55.567608: val_loss -0.7348 
+2024-09-07 18:38:55.568691: Pseudo dice [0.9671, 0.9589, 0.981, 0.9562, 0.8703, 0.9558, 0.987] 
+2024-09-07 18:38:55.569515: Epoch time: 406.85 s 
+2024-09-07 18:38:59.181295:  
+2024-09-07 18:38:59.199934: Epoch 486 
+2024-09-07 18:38:59.202152: Current learning rate: 0.00549 
+2024-09-07 18:46:12.272828: train_loss -0.741 
+2024-09-07 18:46:12.290392: val_loss -0.7673 
+2024-09-07 18:46:12.291991: Pseudo dice [0.9644, 0.9572, 0.9772, 0.9465, 0.8826, 0.9593, 0.9898] 
+2024-09-07 18:46:12.293669: Epoch time: 433.13 s 
+2024-09-07 18:46:16.329551:  
+2024-09-07 18:46:16.331976: Epoch 487 
+2024-09-07 18:46:16.333098: Current learning rate: 0.00548 
+2024-09-07 18:53:10.152207: train_loss -0.7414 
+2024-09-07 18:53:10.160948: val_loss -0.7576 
+2024-09-07 18:53:10.162853: Pseudo dice [0.9662, 0.9588, 0.9817, 0.9482, 0.8553, 0.9654, 0.9881] 
+2024-09-07 18:53:10.164055: Epoch time: 413.83 s 
+2024-09-07 18:53:14.405246:  
+2024-09-07 18:53:14.408451: Epoch 488 
+2024-09-07 18:53:14.409894: Current learning rate: 0.00547 
+2024-09-07 19:00:17.808727: train_loss -0.7296 
+2024-09-07 19:00:17.831557: val_loss -0.7496 
+2024-09-07 19:00:17.834088: Pseudo dice [0.9648, 0.9578, 0.9794, 0.9502, 0.885, 0.969, 0.9915] 
+2024-09-07 19:00:17.845219: Epoch time: 423.46 s 
+2024-09-07 19:00:17.847060: Yayy! New best EMA pseudo Dice: 0.9518 
+2024-09-07 19:00:28.180236:  
+2024-09-07 19:00:28.181479: Epoch 489 
+2024-09-07 19:00:28.182291: Current learning rate: 0.00546 
+2024-09-07 19:07:23.832050: train_loss -0.7505 
+2024-09-07 19:07:23.850753: val_loss -0.7442 
+2024-09-07 19:07:23.852781: Pseudo dice [0.9619, 0.9558, 0.9759, 0.9463, 0.8717, 0.9608, 0.9865] 
+2024-09-07 19:07:23.854842: Epoch time: 415.66 s 
+2024-09-07 19:07:28.939769:  
+2024-09-07 19:07:28.948745: Epoch 490 
+2024-09-07 19:07:28.950205: Current learning rate: 0.00546 
+2024-09-07 19:14:32.753010: train_loss -0.7417 
+2024-09-07 19:14:32.760028: val_loss -0.7518 
+2024-09-07 19:14:32.760997: Pseudo dice [0.9646, 0.958, 0.9768, 0.9521, 0.8804, 0.9698, 0.9902] 
+2024-09-07 19:14:32.762203: Epoch time: 423.82 s 
+2024-09-07 19:14:32.763785: Yayy! New best EMA pseudo Dice: 0.9522 
+2024-09-07 19:14:42.272476:  
+2024-09-07 19:14:42.273767: Epoch 491 
+2024-09-07 19:14:42.274560: Current learning rate: 0.00545 
+2024-09-07 19:22:01.476944: train_loss -0.7443 
+2024-09-07 19:22:01.494510: val_loss -0.7622 
+2024-09-07 19:22:01.496187: Pseudo dice [0.962, 0.9536, 0.9768, 0.9409, 0.8449, 0.9614, 0.9893] 
+2024-09-07 19:22:01.498293: Epoch time: 439.21 s 
+2024-09-07 19:22:06.043861:  
+2024-09-07 19:22:06.045473: Epoch 492 
+2024-09-07 19:22:06.048016: Current learning rate: 0.00544 
+2024-09-07 19:29:18.350727: train_loss -0.716 
+2024-09-07 19:29:18.357026: val_loss -0.7358 
+2024-09-07 19:29:18.359725: Pseudo dice [0.9634, 0.9576, 0.9747, 0.9496, 0.8573, 0.9646, 0.9865] 
+2024-09-07 19:29:18.360833: Epoch time: 432.31 s 
+2024-09-07 19:29:22.649858:  
+2024-09-07 19:29:22.664762: Epoch 493 
+2024-09-07 19:29:22.665743: Current learning rate: 0.00543 
+2024-09-07 19:36:19.153601: train_loss -0.7489 
+2024-09-07 19:36:19.163612: val_loss -0.7968 
+2024-09-07 19:36:19.165341: Pseudo dice [0.9634, 0.9541, 0.9773, 0.95, 0.8702, 0.9692, 0.992] 
+2024-09-07 19:36:19.166468: Epoch time: 416.51 s 
+2024-09-07 19:36:25.285217:  
+2024-09-07 19:36:25.286734: Epoch 494 
+2024-09-07 19:36:25.287726: Current learning rate: 0.00542 
+2024-09-07 19:43:39.693748: train_loss -0.7323 
+2024-09-07 19:43:39.701944: val_loss -0.713 
+2024-09-07 19:43:39.703077: Pseudo dice [0.9611, 0.9565, 0.9754, 0.9488, 0.8841, 0.9586, 0.9912] 
+2024-09-07 19:43:39.705238: Epoch time: 434.43 s 
+2024-09-07 19:43:44.911214:  
+2024-09-07 19:43:44.920810: Epoch 495 
+2024-09-07 19:43:44.921809: Current learning rate: 0.00541 
+2024-09-07 19:50:40.551356: train_loss -0.7162 
+2024-09-07 19:50:40.564754: val_loss -0.6874 
+2024-09-07 19:50:40.566417: Pseudo dice [0.9604, 0.9548, 0.9649, 0.94, 0.8352, 0.9517, 0.984] 
+2024-09-07 19:50:40.568866: Epoch time: 415.7 s 
+2024-09-07 19:50:44.831706:  
+2024-09-07 19:50:44.848766: Epoch 496 
+2024-09-07 19:50:44.851297: Current learning rate: 0.0054 
+2024-09-07 19:58:18.256136: train_loss -0.7225 
+2024-09-07 19:58:18.259226: val_loss -0.6992 
+2024-09-07 19:58:18.261594: Pseudo dice [0.9627, 0.9537, 0.9766, 0.948, 0.8737, 0.9335, 0.984] 
+2024-09-07 19:58:18.263122: Epoch time: 453.41 s 
+2024-09-07 19:58:23.402942:  
+2024-09-07 19:58:23.418317: Epoch 497 
+2024-09-07 19:58:23.420514: Current learning rate: 0.00539 
+2024-09-07 20:05:38.436595: train_loss -0.7293 
+2024-09-07 20:05:38.452387: val_loss -0.7487 
+2024-09-07 20:05:38.454169: Pseudo dice [0.9608, 0.9532, 0.977, 0.9459, 0.8606, 0.963, 0.9904] 
+2024-09-07 20:05:38.455134: Epoch time: 435.04 s 
+2024-09-07 20:05:41.628126:  
+2024-09-07 20:05:41.634161: Epoch 498 
+2024-09-07 20:05:41.635503: Current learning rate: 0.00538 
+2024-09-07 20:12:35.202531: train_loss -0.7269 
+2024-09-07 20:12:35.204808: val_loss -0.7416 
+2024-09-07 20:12:35.206339: Pseudo dice [0.9631, 0.9561, 0.978, 0.95, 0.8678, 0.9666, 0.9891] 
+2024-09-07 20:12:35.207552: Epoch time: 413.58 s 
+2024-09-07 20:12:40.688826:  
+2024-09-07 20:12:40.695918: Epoch 499 
+2024-09-07 20:12:40.699225: Current learning rate: 0.00537 
+2024-09-07 20:19:48.504694: train_loss -0.7553 
+2024-09-07 20:19:48.521519: val_loss -0.7095 
+2024-09-07 20:19:48.523088: Pseudo dice [0.9614, 0.9562, 0.9748, 0.9441, 0.8605, 0.9598, 0.9852] 
+2024-09-07 20:19:48.524008: Epoch time: 427.83 s 

nnUNetTrainer__nnUNetResEncUNetLPlans__3d_fullres/fold_1/logs/training_log_2024_9_7_22_04_47.txt

@@ -0,0 +1,385 @@
+
+#######################################################################
+Please cite the following paper when using nnU-Net:
+Isensee, F., Jaeger, P. F., Kohl, S. A., Petersen, J., & Maier-Hein, K. H. (2021). nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation. Nature methods, 18(2), 203-211.
+#######################################################################
+ 
+2024-09-07 22:04:50.674146: Using torch.compile... 
+2024-09-07 22:04:57.750741: do_dummy_2d_data_aug: False 
+2024-09-07 22:04:57.788201: Using splits from existing split file: <workspace>/data2/nnUNet_preprocessed/Dataset611_Tissue/splits_final.json 
+2024-09-07 22:04:57.819455: The split file contains 5 splits. 
+2024-09-07 22:04:57.820192: Desired fold for training: 1 
+2024-09-07 22:04:57.820838: This split has 1325 training and 331 validation cases. 
+
+This is the configuration used by this training:
+Configuration name: 3d_fullres
+ {'data_identifier': 'nnUNetPlans_3d_fullres', 'preprocessor_name': 'DefaultPreprocessor', 'batch_size': 2, 'patch_size': [192, 192, 192], 'median_image_size_in_voxels': [250.0, 257.0, 260.0], 'spacing': [1.5, 1.5, 1.5], 'normalization_schemes': ['CTNormalization'], 'use_mask_for_norm': [False], 'resampling_fn_data': 'resample_data_or_seg_to_shape', 'resampling_fn_seg': 'resample_data_or_seg_to_shape', 'resampling_fn_data_kwargs': {'is_seg': False, 'order': 3, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_seg_kwargs': {'is_seg': True, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'resampling_fn_probabilities': 'resample_data_or_seg_to_shape', 'resampling_fn_probabilities_kwargs': {'is_seg': False, 'order': 1, 'order_z': 0, 'force_separate_z': None}, 'architecture': {'network_class_name': 'dynamic_network_architectures.architectures.unet.ResidualEncoderUNet', 'arch_kwargs': {'n_stages': 6, 'features_per_stage': [32, 64, 128, 256, 320, 320], 'conv_op': 'torch.nn.modules.conv.Conv3d', 'kernel_sizes': [[3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3], [3, 3, 3]], 'strides': [[1, 1, 1], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2], [2, 2, 2]], 'n_blocks_per_stage': [1, 3, 4, 6, 6, 6], 'n_conv_per_stage_decoder': [1, 1, 1, 1, 1], 'conv_bias': True, 'norm_op': 'torch.nn.modules.instancenorm.InstanceNorm3d', 'norm_op_kwargs': {'eps': 1e-05, 'affine': True}, 'dropout_op': None, 'dropout_op_kwargs': None, 'nonlin': 'torch.nn.LeakyReLU', 'nonlin_kwargs': {'inplace': True}, 'deep_supervision': True}, '_kw_requires_import': ['conv_op', 'norm_op', 'dropout_op', 'nonlin']}, 'batch_dice': False} 
+ 
+These are the global plan.json settings:
+ {'dataset_name': 'Dataset611_Tissue', 'plans_name': 'nnUNetResEncUNetLPlans', 'original_median_spacing_after_transp': [1.5, 1.5, 1.5], 'original_median_shape_after_transp': [170, 309, 309], 'image_reader_writer': 'SimpleITKIO', 'transpose_forward': [0, 1, 2], 'transpose_backward': [0, 1, 2], 'experiment_planner_used': 'nnUNetPlannerResEncL', 'label_manager': 'LabelManager', 'foreground_intensity_properties_per_channel': {'0': {'max': 9580.0, 'mean': -115.16296809491125, 'median': -61.0, 'min': -3048.0, 'percentile_00_5': -948.0, 'percentile_99_5': 816.0, 'std': 324.74287592532374}}} 
+ 
+2024-09-07 22:05:11.427322: Unable to plot network architecture: nnUNet_compile is enabled! 
+2024-09-07 22:05:12.598591:  
+2024-09-07 22:05:12.608421: Epoch 500 
+2024-09-07 22:05:12.610506: Current learning rate: 0.00536 
+2024-09-07 22:21:04.260250: train_loss -0.7398 
+2024-09-07 22:21:04.263559: val_loss -0.7209 
+2024-09-07 22:21:04.264463: Pseudo dice [0.9621, 0.9562, 0.9767, 0.9483, 0.8616, 0.9663, 0.9913] 
+2024-09-07 22:21:04.265229: Epoch time: 951.67 s 
+2024-09-07 22:21:06.024849:  
+2024-09-07 22:21:06.025752: Epoch 501 
+2024-09-07 22:21:06.026429: Current learning rate: 0.00535 
+2024-09-07 22:27:01.595051: train_loss -0.7322 
+2024-09-07 22:27:01.610305: val_loss -0.7543 
+2024-09-07 22:27:01.611291: Pseudo dice [0.9664, 0.9573, 0.9804, 0.9528, 0.8841, 0.9482, 0.9869] 
+2024-09-07 22:27:01.612160: Epoch time: 355.57 s 
+2024-09-07 22:27:05.025893:  
+2024-09-07 22:27:05.036460: Epoch 502 
+2024-09-07 22:27:05.037537: Current learning rate: 0.00534 
+2024-09-07 22:32:54.964900: train_loss -0.7431 
+2024-09-07 22:32:54.976601: val_loss -0.7513 
+2024-09-07 22:32:54.977866: Pseudo dice [0.9629, 0.9557, 0.9766, 0.9519, 0.8571, 0.9663, 0.9919] 
+2024-09-07 22:32:54.979050: Epoch time: 349.94 s 
+2024-09-07 22:32:59.705626:  
+2024-09-07 22:32:59.708675: Epoch 503 
+2024-09-07 22:32:59.709657: Current learning rate: 0.00533 
+2024-09-07 22:38:27.674155: train_loss -0.7444 
+2024-09-07 22:38:27.675317: val_loss -0.7549 
+2024-09-07 22:38:27.676009: Pseudo dice [0.9654, 0.957, 0.9789, 0.95, 0.8875, 0.9723, 0.9918] 
+2024-09-07 22:38:27.676690: Epoch time: 327.97 s 
+2024-09-07 22:38:33.919375:  
+2024-09-07 22:38:33.920747: Epoch 504 
+2024-09-07 22:38:33.921581: Current learning rate: 0.00532 
+2024-09-07 22:44:03.917117: train_loss -0.7518 
+2024-09-07 22:44:03.929006: val_loss -0.7331 
+2024-09-07 22:44:03.929857: Pseudo dice [0.9653, 0.9578, 0.9778, 0.9478, 0.8685, 0.9683, 0.9885] 
+2024-09-07 22:44:03.930684: Epoch time: 330.0 s 
+2024-09-07 22:44:06.844565:  
+2024-09-07 22:44:06.845719: Epoch 505 
+2024-09-07 22:44:06.846606: Current learning rate: 0.00531 
+2024-09-07 22:49:36.800798: train_loss -0.7424 
+2024-09-07 22:49:36.816910: val_loss -0.7577 
+2024-09-07 22:49:36.818784: Pseudo dice [0.9646, 0.9562, 0.9783, 0.9502, 0.8739, 0.9717, 0.9907] 
+2024-09-07 22:49:36.820037: Epoch time: 329.96 s 
+2024-09-07 22:49:36.821064: Yayy! New best EMA pseudo Dice: 0.9522 
+2024-09-07 22:49:44.642648:  
+2024-09-07 22:49:44.644330: Epoch 506 
+2024-09-07 22:49:44.646209: Current learning rate: 0.0053 
+2024-09-07 22:55:08.754494: train_loss -0.7401 
+2024-09-07 22:55:08.771585: val_loss -0.7548 
+2024-09-07 22:55:08.772772: Pseudo dice [0.9659, 0.959, 0.9802, 0.9524, 0.8752, 0.9707, 0.9911] 
+2024-09-07 22:55:08.773713: Epoch time: 324.12 s 
+2024-09-07 22:55:08.774830: Yayy! New best EMA pseudo Dice: 0.9526 
+2024-09-07 22:55:15.407771:  
+2024-09-07 22:55:15.408896: Epoch 507 
+2024-09-07 22:55:15.409642: Current learning rate: 0.00529 
+2024-09-07 23:00:39.652704: train_loss -0.754 
+2024-09-07 23:00:39.655074: val_loss -0.7342 
+2024-09-07 23:00:39.656199: Pseudo dice [0.963, 0.9564, 0.9763, 0.953, 0.8771, 0.9629, 0.9893] 
+2024-09-07 23:00:39.657193: Epoch time: 324.25 s 
+2024-09-07 23:00:39.658129: Yayy! New best EMA pseudo Dice: 0.9528 
+2024-09-07 23:00:46.556384:  
+2024-09-07 23:00:46.557528: Epoch 508 
+2024-09-07 23:00:46.558377: Current learning rate: 0.00528 
+2024-09-07 23:06:12.891781: train_loss -0.7333 
+2024-09-07 23:06:12.906415: val_loss -0.7385 
+2024-09-07 23:06:12.907601: Pseudo dice [0.9623, 0.955, 0.9775, 0.9495, 0.8686, 0.9657, 0.9893] 
+2024-09-07 23:06:12.908571: Epoch time: 326.34 s 
+2024-09-07 23:06:15.618807:  
+2024-09-07 23:06:15.620229: Epoch 509 
+2024-09-07 23:06:15.621056: Current learning rate: 0.00527 
+2024-09-07 23:11:33.792571: train_loss -0.7277 
+2024-09-07 23:11:33.810715: val_loss -0.7356 
+2024-09-07 23:11:33.812034: Pseudo dice [0.9616, 0.9562, 0.9722, 0.9455, 0.8789, 0.9714, 0.9914] 
+2024-09-07 23:11:33.813629: Epoch time: 318.18 s 
+2024-09-07 23:11:33.814553: Yayy! New best EMA pseudo Dice: 0.9529 
+2024-09-07 23:11:39.533535:  
+2024-09-07 23:11:39.534552: Epoch 510 
+2024-09-07 23:11:39.535235: Current learning rate: 0.00526 
+2024-09-07 23:16:55.929312: train_loss -0.742 
+2024-09-07 23:16:55.945905: val_loss -0.7288 
+2024-09-07 23:16:55.946886: Pseudo dice [0.9621, 0.9565, 0.9755, 0.9465, 0.8699, 0.9684, 0.9893] 
+2024-09-07 23:16:55.947719: Epoch time: 316.4 s 
+2024-09-07 23:16:59.511015:  
+2024-09-07 23:16:59.512851: Epoch 511 
+2024-09-07 23:16:59.514108: Current learning rate: 0.00525 
+2024-09-07 23:22:36.673199: train_loss -0.7521 
+2024-09-07 23:22:36.676011: val_loss -0.7665 
+2024-09-07 23:22:36.677162: Pseudo dice [0.9642, 0.9569, 0.9784, 0.9499, 0.8657, 0.969, 0.9902] 
+2024-09-07 23:22:36.678409: Epoch time: 337.17 s 
+2024-09-07 23:22:36.679560: Yayy! New best EMA pseudo Dice: 0.9529 
+2024-09-07 23:22:41.167937:  
+2024-09-07 23:22:41.170083: Epoch 512 
+2024-09-07 23:22:41.170802: Current learning rate: 0.00524 
+2024-09-07 23:28:03.595278: train_loss -0.7398 
+2024-09-07 23:28:03.609780: val_loss -0.7419 
+2024-09-07 23:28:03.610916: Pseudo dice [0.9646, 0.9554, 0.9795, 0.9525, 0.8626, 0.9684, 0.9889] 
+2024-09-07 23:28:03.611777: Epoch time: 322.43 s 
+2024-09-07 23:28:03.613013: Yayy! New best EMA pseudo Dice: 0.9529 
+2024-09-07 23:28:08.306090:  
+2024-09-07 23:28:08.307370: Epoch 513 
+2024-09-07 23:28:08.308791: Current learning rate: 0.00523 
+2024-09-07 23:33:36.025498: train_loss -0.7332 
+2024-09-07 23:33:36.042332: val_loss -0.737 
+2024-09-07 23:33:36.043935: Pseudo dice [0.9654, 0.9579, 0.9806, 0.9505, 0.8779, 0.964, 0.9888] 
+2024-09-07 23:33:36.048609: Epoch time: 327.72 s 
+2024-09-07 23:33:36.050602: Yayy! New best EMA pseudo Dice: 0.9531 
+2024-09-07 23:33:41.603685:  
+2024-09-07 23:33:41.605055: Epoch 514 
+2024-09-07 23:33:41.606210: Current learning rate: 0.00522 
+2024-09-07 23:39:32.964052: train_loss -0.7403 
+2024-09-07 23:39:32.967588: val_loss -0.7349 
+2024-09-07 23:39:32.969013: Pseudo dice [0.9628, 0.9573, 0.9766, 0.9495, 0.8624, 0.9722, 0.9916] 
+2024-09-07 23:39:32.969997: Epoch time: 351.36 s 
+2024-09-07 23:39:32.971043: Yayy! New best EMA pseudo Dice: 0.9531 
+2024-09-07 23:39:39.154284:  
+2024-09-07 23:39:39.155391: Epoch 515 
+2024-09-07 23:39:39.156266: Current learning rate: 0.00521 
+2024-09-07 23:45:12.937903: train_loss -0.7412 
+2024-09-07 23:45:12.947747: val_loss -0.7155 
+2024-09-07 23:45:12.948617: Pseudo dice [0.9678, 0.9578, 0.9813, 0.9505, 0.8112, 0.9664, 0.9823] 
+2024-09-07 23:45:12.949275: Epoch time: 333.79 s 
+2024-09-07 23:45:16.321935:  
+2024-09-07 23:45:16.337783: Epoch 516 
+2024-09-07 23:45:16.339195: Current learning rate: 0.0052 
+2024-09-07 23:52:10.594391: train_loss -0.7383 
+2024-09-07 23:52:10.608208: val_loss -0.7503 
+2024-09-07 23:52:10.609088: Pseudo dice [0.9628, 0.9564, 0.9791, 0.9502, 0.8518, 0.9668, 0.9889] 
+2024-09-07 23:52:10.609827: Epoch time: 414.28 s 
+2024-09-07 23:52:14.357829:  
+2024-09-07 23:52:14.373017: Epoch 517 
+2024-09-07 23:52:14.374698: Current learning rate: 0.00519 
+2024-09-07 23:58:58.745918: train_loss -0.755 
+2024-09-07 23:58:58.756342: val_loss -0.7609 
+2024-09-07 23:58:58.757741: Pseudo dice [0.9654, 0.9558, 0.9776, 0.9489, 0.8694, 0.9565, 0.9898] 
+2024-09-07 23:58:58.758785: Epoch time: 404.4 s 
+2024-09-07 23:59:02.348619:  
+2024-09-07 23:59:02.361170: Epoch 518 
+2024-09-07 23:59:02.362031: Current learning rate: 0.00518 
+2024-09-08 00:05:36.059265: train_loss -0.7263 
+2024-09-08 00:05:36.073974: val_loss -0.7102 
+2024-09-08 00:05:36.075844: Pseudo dice [0.9663, 0.96, 0.9798, 0.9495, 0.8797, 0.9728, 0.9917] 
+2024-09-08 00:05:36.077649: Epoch time: 393.72 s 
+2024-09-08 00:05:38.927080:  
+2024-09-08 00:05:38.936911: Epoch 519 
+2024-09-08 00:05:38.937690: Current learning rate: 0.00518 
+2024-09-08 00:11:09.908194: train_loss -0.7547 
+2024-09-08 00:11:09.916192: val_loss -0.7732 
+2024-09-08 00:11:09.918207: Pseudo dice [0.9647, 0.9575, 0.9774, 0.9479, 0.8344, 0.9717, 0.9894] 
+2024-09-08 00:11:09.920044: Epoch time: 330.99 s 
+2024-09-08 00:11:13.167482:  
+2024-09-08 00:11:13.169036: Epoch 520 
+2024-09-08 00:11:13.170200: Current learning rate: 0.00517 
+2024-09-08 00:17:04.781944: train_loss -0.7514 
+2024-09-08 00:17:04.785143: val_loss -0.7299 
+2024-09-08 00:17:04.786303: Pseudo dice [0.9668, 0.9598, 0.9814, 0.9529, 0.8306, 0.9691, 0.9853] 
+2024-09-08 00:17:04.787292: Epoch time: 351.62 s 
+2024-09-08 00:17:08.020002:  
+2024-09-08 00:17:08.021100: Epoch 521 
+2024-09-08 00:17:08.021905: Current learning rate: 0.00516 
+2024-09-08 00:22:44.999535: train_loss -0.7378 
+2024-09-08 00:22:45.005912: val_loss -0.7616 
+2024-09-08 00:22:45.007336: Pseudo dice [0.9669, 0.9595, 0.9812, 0.9537, 0.8836, 0.9623, 0.9897] 
+2024-09-08 00:22:45.008552: Epoch time: 336.99 s 
+2024-09-08 00:22:48.521873:  
+2024-09-08 00:22:48.536937: Epoch 522 
+2024-09-08 00:22:48.537869: Current learning rate: 0.00515 
+2024-09-08 00:28:27.182241: train_loss -0.7467 
+2024-09-08 00:28:27.197551: val_loss -0.7577 
+2024-09-08 00:28:27.198842: Pseudo dice [0.9645, 0.9595, 0.9763, 0.953, 0.8651, 0.9704, 0.9893] 
+2024-09-08 00:28:27.199883: Epoch time: 338.67 s 
+2024-09-08 00:28:31.277493:  
+2024-09-08 00:28:31.278579: Epoch 523 
+2024-09-08 00:28:31.279397: Current learning rate: 0.00514 
+2024-09-08 00:34:11.171606: train_loss -0.7492 
+2024-09-08 00:34:11.194882: val_loss -0.7649 
+2024-09-08 00:34:11.196158: Pseudo dice [0.9634, 0.9588, 0.9747, 0.9474, 0.8871, 0.9706, 0.9911] 
+2024-09-08 00:34:11.196872: Epoch time: 339.9 s 
+2024-09-08 00:34:14.613741:  
+2024-09-08 00:34:14.627441: Epoch 524 
+2024-09-08 00:34:14.629097: Current learning rate: 0.00513 
+2024-09-08 00:40:02.588592: train_loss -0.7472 
+2024-09-08 00:40:02.590621: val_loss -0.7029 
+2024-09-08 00:40:02.592004: Pseudo dice [0.9623, 0.9569, 0.9759, 0.9507, 0.8606, 0.9545, 0.9818] 
+2024-09-08 00:40:02.592931: Epoch time: 347.98 s 
+2024-09-08 00:40:05.773774:  
+2024-09-08 00:40:05.774814: Epoch 525 
+2024-09-08 00:40:05.775472: Current learning rate: 0.00512 
+2024-09-08 00:45:27.993200: train_loss -0.7445 
+2024-09-08 00:45:28.012009: val_loss -0.7702 
+2024-09-08 00:45:28.013687: Pseudo dice [0.9661, 0.9589, 0.9807, 0.9503, 0.8932, 0.9509, 0.9879] 
+2024-09-08 00:45:28.014740: Epoch time: 322.22 s 
+2024-09-08 00:45:32.190089:  
+2024-09-08 00:45:32.191692: Epoch 526 
+2024-09-08 00:45:32.193161: Current learning rate: 0.00511 
+2024-09-08 00:50:56.829152: train_loss -0.7548 
+2024-09-08 00:50:56.847797: val_loss -0.7534 
+2024-09-08 00:50:56.849786: Pseudo dice [0.9662, 0.9575, 0.9801, 0.9499, 0.886, 0.9725, 0.9908] 
+2024-09-08 00:50:56.851396: Epoch time: 324.64 s 
+2024-09-08 00:50:56.852328: Yayy! New best EMA pseudo Dice: 0.9533 
+2024-09-08 00:51:05.249813:  
+2024-09-08 00:51:05.250811: Epoch 527 
+2024-09-08 00:51:05.251546: Current learning rate: 0.0051 
+2024-09-08 00:56:31.967358: train_loss -0.7288 
+2024-09-08 00:56:31.970717: val_loss -0.7245 
+2024-09-08 00:56:31.971931: Pseudo dice [0.9646, 0.9574, 0.9802, 0.9523, 0.8378, 0.9684, 0.9848] 
+2024-09-08 00:56:31.972950: Epoch time: 326.72 s 
+2024-09-08 00:56:36.443444:  
+2024-09-08 00:56:36.444331: Epoch 528 
+2024-09-08 00:56:36.444964: Current learning rate: 0.00509 
+2024-09-08 01:02:30.080415: train_loss -0.7467 
+2024-09-08 01:02:30.082865: val_loss -0.7411 
+2024-09-08 01:02:30.084260: Pseudo dice [0.9612, 0.9535, 0.9728, 0.9488, 0.8687, 0.971, 0.9914] 
+2024-09-08 01:02:30.085149: Epoch time: 353.64 s 
+2024-09-08 01:02:33.965918:  
+2024-09-08 01:02:33.969100: Epoch 529 
+2024-09-08 01:02:33.970279: Current learning rate: 0.00508 
+2024-09-08 01:08:03.797636: train_loss -0.7532 
+2024-09-08 01:08:03.814926: val_loss -0.7286 
+2024-09-08 01:08:03.816105: Pseudo dice [0.9653, 0.9599, 0.9787, 0.953, 0.8716, 0.9703, 0.9913] 
+2024-09-08 01:08:03.816940: Epoch time: 329.84 s 
+2024-09-08 01:08:07.616443:  
+2024-09-08 01:08:07.628764: Epoch 530 
+2024-09-08 01:08:07.629886: Current learning rate: 0.00507 
+2024-09-08 01:13:59.198542: train_loss -0.7349 
+2024-09-08 01:13:59.209613: val_loss -0.7768 
+2024-09-08 01:13:59.210567: Pseudo dice [0.9666, 0.955, 0.9803, 0.9488, 0.881, 0.9705, 0.9921] 
+2024-09-08 01:13:59.211408: Epoch time: 351.59 s 
+2024-09-08 01:13:59.212460: Yayy! New best EMA pseudo Dice: 0.9535 
+2024-09-08 01:14:06.092178:  
+2024-09-08 01:14:06.093483: Epoch 531 
+2024-09-08 01:14:06.094298: Current learning rate: 0.00506 
+2024-09-08 01:19:41.145960: train_loss -0.7287 
+2024-09-08 01:19:41.149267: val_loss -0.737 
+2024-09-08 01:19:41.150827: Pseudo dice [0.9643, 0.9587, 0.9748, 0.9525, 0.8691, 0.9661, 0.9885] 
+2024-09-08 01:19:41.152224: Epoch time: 335.06 s 
+2024-09-08 01:19:44.278082:  
+2024-09-08 01:19:44.279293: Epoch 532 
+2024-09-08 01:19:44.280030: Current learning rate: 0.00505 
+2024-09-08 01:25:20.521732: train_loss -0.7188 
+2024-09-08 01:25:20.537614: val_loss -0.7661 
+2024-09-08 01:25:20.538392: Pseudo dice [0.9637, 0.9568, 0.9773, 0.953, 0.8634, 0.9576, 0.9862] 
+2024-09-08 01:25:20.539106: Epoch time: 336.26 s 
+2024-09-08 01:25:24.300575:  
+2024-09-08 01:25:24.302175: Epoch 533 
+2024-09-08 01:25:24.303513: Current learning rate: 0.00504 
+2024-09-08 01:31:42.191687: train_loss -0.7531 
+2024-09-08 01:31:42.211203: val_loss -0.7741 
+2024-09-08 01:31:42.212934: Pseudo dice [0.962, 0.9565, 0.9734, 0.9511, 0.8854, 0.9722, 0.9917] 
+2024-09-08 01:31:42.214158: Epoch time: 377.92 s 
+2024-09-08 01:31:42.215092: Yayy! New best EMA pseudo Dice: 0.9535 
+2024-09-08 01:31:48.820838:  
+2024-09-08 01:31:48.821676: Epoch 534 
+2024-09-08 01:31:48.822562: Current learning rate: 0.00503 
+2024-09-08 01:37:31.884438: train_loss -0.7385 
+2024-09-08 01:37:31.886100: val_loss -0.7746 
+2024-09-08 01:37:31.887375: Pseudo dice [0.9626, 0.9575, 0.9741, 0.9507, 0.8849, 0.9705, 0.9909] 
+2024-09-08 01:37:31.888183: Epoch time: 343.07 s 
+2024-09-08 01:37:31.888944: Yayy! New best EMA pseudo Dice: 0.9538 
+2024-09-08 01:37:37.881222:  
+2024-09-08 01:37:37.882759: Epoch 535 
+2024-09-08 01:37:37.883676: Current learning rate: 0.00502 
+2024-09-08 01:43:47.670296: train_loss -0.7477 
+2024-09-08 01:43:47.678148: val_loss -0.7407 
+2024-09-08 01:43:47.680726: Pseudo dice [0.9656, 0.9593, 0.9777, 0.9554, 0.8878, 0.9555, 0.9891] 
+2024-09-08 01:43:47.694768: Epoch time: 369.78 s 
+2024-09-08 01:43:47.696836: Yayy! New best EMA pseudo Dice: 0.954 
+2024-09-08 01:43:53.887015:  
+2024-09-08 01:43:53.888157: Epoch 536 
+2024-09-08 01:43:53.888961: Current learning rate: 0.00501 
+2024-09-08 01:49:35.421767: train_loss -0.7402 
+2024-09-08 01:49:35.427513: val_loss -0.7446 
+2024-09-08 01:49:35.428667: Pseudo dice [0.9652, 0.958, 0.9811, 0.9497, 0.8645, 0.9718, 0.9884] 
+2024-09-08 01:49:35.429588: Epoch time: 341.54 s 
+2024-09-08 01:49:35.430348: Yayy! New best EMA pseudo Dice: 0.954 
+2024-09-08 01:49:43.140928:  
+2024-09-08 01:49:43.152977: Epoch 537 
+2024-09-08 01:49:43.155103: Current learning rate: 0.005 
+2024-09-08 01:55:16.378089: train_loss -0.7392 
+2024-09-08 01:55:16.381190: val_loss -0.7069 
+2024-09-08 01:55:16.383257: Pseudo dice [0.9621, 0.958, 0.975, 0.9532, 0.8808, 0.9359, 0.9828] 
+2024-09-08 01:55:16.384174: Epoch time: 333.24 s 
+2024-09-08 01:55:19.113398:  
+2024-09-08 01:55:19.114724: Epoch 538 
+2024-09-08 01:55:19.115589: Current learning rate: 0.00499 
+2024-09-08 02:00:51.115720: train_loss -0.7352 
+2024-09-08 02:00:51.118815: val_loss -0.7647 
+2024-09-08 02:00:51.119727: Pseudo dice [0.9628, 0.9556, 0.976, 0.9493, 0.8605, 0.9595, 0.9869] 
+2024-09-08 02:00:51.120524: Epoch time: 332.01 s 
+2024-09-08 02:00:55.032301:  
+2024-09-08 02:00:55.033691: Epoch 539 
+2024-09-08 02:00:55.034566: Current learning rate: 0.00498 
+2024-09-08 02:06:57.293925: train_loss -0.7316 
+2024-09-08 02:06:57.309182: val_loss -0.7042 
+2024-09-08 02:06:57.309959: Pseudo dice [0.9615, 0.9546, 0.9737, 0.9498, 0.8474, 0.9693, 0.9888] 
+2024-09-08 02:06:57.310859: Epoch time: 362.27 s 
+2024-09-08 02:07:02.363613:  
+2024-09-08 02:07:02.365546: Epoch 540 
+2024-09-08 02:07:02.366598: Current learning rate: 0.00497 
+2024-09-08 02:12:33.163837: train_loss -0.7304 
+2024-09-08 02:12:33.176619: val_loss -0.7435 
+2024-09-08 02:12:33.177656: Pseudo dice [0.9618, 0.9575, 0.9756, 0.9477, 0.8719, 0.9708, 0.9888] 
+2024-09-08 02:12:33.178734: Epoch time: 330.81 s 
+2024-09-08 02:12:36.840210:  
+2024-09-08 02:12:36.842304: Epoch 541 
+2024-09-08 02:12:36.843409: Current learning rate: 0.00496 
+2024-09-08 02:18:29.393152: train_loss -0.7537 
+2024-09-08 02:18:29.407128: val_loss -0.7473 
+2024-09-08 02:18:29.409021: Pseudo dice [0.9603, 0.9567, 0.9724, 0.9456, 0.841, 0.9701, 0.9873] 
+2024-09-08 02:18:29.410938: Epoch time: 352.57 s 
+2024-09-08 02:18:33.340688:  
+2024-09-08 02:18:33.341793: Epoch 542 
+2024-09-08 02:18:33.342881: Current learning rate: 0.00495 
+2024-09-08 02:24:15.424063: train_loss -0.74 
+2024-09-08 02:24:15.452756: val_loss -0.7629 
+2024-09-08 02:24:15.453601: Pseudo dice [0.9639, 0.9561, 0.9772, 0.9489, 0.8715, 0.9712, 0.991] 
+2024-09-08 02:24:15.454668: Epoch time: 342.09 s 
+2024-09-08 02:24:19.456515:  
+2024-09-08 02:24:19.457755: Epoch 543 
+2024-09-08 02:24:19.459323: Current learning rate: 0.00494 
+2024-09-08 02:30:12.622942: train_loss -0.7177 
+2024-09-08 02:30:12.638146: val_loss -0.7834 
+2024-09-08 02:30:12.639125: Pseudo dice [0.9662, 0.9569, 0.9804, 0.9523, 0.8934, 0.9736, 0.9928] 
+2024-09-08 02:30:12.641610: Epoch time: 353.17 s 
+2024-09-08 02:30:15.984111:  
+2024-09-08 02:30:15.996773: Epoch 544 
+2024-09-08 02:30:15.998461: Current learning rate: 0.00493 
+2024-09-08 02:36:00.920480: train_loss -0.7251 
+2024-09-08 02:36:00.938487: val_loss -0.7268 
+2024-09-08 02:36:00.939322: Pseudo dice [0.9634, 0.9596, 0.9786, 0.954, 0.8527, 0.9688, 0.9908] 
+2024-09-08 02:36:00.939994: Epoch time: 344.94 s 
+2024-09-08 02:36:04.220381:  
+2024-09-08 02:36:04.224324: Epoch 545 
+2024-09-08 02:36:04.225306: Current learning rate: 0.00492 
+2024-09-08 02:41:59.952780: train_loss -0.7314 
+2024-09-08 02:41:59.969000: val_loss -0.7253 
+2024-09-08 02:41:59.969958: Pseudo dice [0.9657, 0.959, 0.9775, 0.9467, 0.8669, 0.9649, 0.991] 
+2024-09-08 02:41:59.970735: Epoch time: 355.74 s 
+2024-09-08 02:42:02.733647:  
+2024-09-08 02:42:02.734620: Epoch 546 
+2024-09-08 02:42:02.735401: Current learning rate: 0.00491 
+2024-09-08 02:48:38.178225: train_loss -0.7495 
+2024-09-08 02:48:38.180454: val_loss -0.7448 
+2024-09-08 02:48:38.181200: Pseudo dice [0.9635, 0.9568, 0.9765, 0.9487, 0.8596, 0.9724, 0.9897] 
+2024-09-08 02:48:38.181846: Epoch time: 395.45 s 
+2024-09-08 02:48:43.200014:  
+2024-09-08 02:48:43.207662: Epoch 547 
+2024-09-08 02:48:43.208690: Current learning rate: 0.0049 
+2024-09-08 02:54:54.700977: train_loss -0.751 
+2024-09-08 02:54:54.718369: val_loss -0.7561 
+2024-09-08 02:54:54.719600: Pseudo dice [0.9634, 0.955, 0.9781, 0.9453, 0.8582, 0.9654, 0.9899] 
+2024-09-08 02:54:54.720823: Epoch time: 371.51 s 
+2024-09-08 02:54:59.948493:  
+2024-09-08 02:54:59.949560: Epoch 548 
+2024-09-08 02:54:59.950268: Current learning rate: 0.00489 
+2024-09-08 03:01:42.142303: train_loss -0.7332 
+2024-09-08 03:01:42.145561: val_loss -0.7559 
+2024-09-08 03:01:42.146505: Pseudo dice [0.9617, 0.9571, 0.9767, 0.9435, 0.8759, 0.9689, 0.9899] 
+2024-09-08 03:01:42.147262: Epoch time: 402.17 s 
+2024-09-08 03:01:46.628851:  
+2024-09-08 03:01:46.629889: Epoch 549 
+2024-09-08 03:01:46.630774: Current learning rate: 0.00488 
+2024-09-08 03:08:21.165940: train_loss -0.7236 
+2024-09-08 03:08:21.181121: val_loss -0.7404 
+2024-09-08 03:08:21.182082: Pseudo dice [0.9604, 0.9555, 0.9765, 0.947, 0.8472, 0.9569, 0.9868] 
+2024-09-08 03:08:21.182888: Epoch time: 394.54 s