OptixLog SDK Overview

The OptixLog SDK is a Python library designed to seamlessly integrate logging and experiment tracking into photonic simulations and hardware testing workflows. It provides a simple, powerful API for logging metrics, images, files, and other artifacts from your simulations and experiments.

What is OptixLog SDK?

OptixLog SDK is a client library that connects your Python code to the OptixLog platform, enabling you to:

Track experiments: Log metrics, parameters, and results from simulations
Organize runs: Group related experiments into projects and runs
Visualize results: Upload images, plots, and data files for easy viewing
Collaborate: Share results with team members through the web dashboard
Automate workflows: Integrate with MPI, simulation frameworks, and hardware test equipment
Query data: Programmatically access runs, metrics, and artifacts
Compare runs: Analyze differences between experiments

Key Features

Zero Boilerplate Logging

Log plots and visualizations with one line of code:

log_matplotlib(): Log matplotlib figures directly (no PIL conversion needed!)
log_plot(): Create and log plots from data in one call
log_array_as_image(): Convert numpy arrays to heatmaps automatically
log_histogram(): Create and log histograms easily
log_scatter(): Log scatter plots with minimal code
log_multiple_plots(): Compare multiple series on one plot

Automatic MPI Support

The SDK automatically detects MPI environments and handles master/worker process coordination:

Automatic detection of OpenMPI, Intel MPI, Microsoft MPI, and mpi4py
Master process logging: Only the master process logs to avoid duplicates
Worker process awareness: Worker processes are initialized but skip logging
Synchronization methods: Built-in barrier and broadcast functions

Context Manager Support

Use Python’s with statement for clean, automatic resource management:


with optixlog.run("my_experiment") as client:
    client.log(step=0, loss=0.5)
    # Automatic cleanup when done!

Input Validation

Automatic validation catches common errors:

NaN/Inf detection: Prevents logging invalid numeric values
Type checking: Ensures correct data types
File validation: Verifies files exist before upload
Image validation: Checks image format and size

Return Values

Every logging method returns useful information:

Success status: Know if logging succeeded
URLs: Get direct links to uploaded artifacts
Metadata: Access file sizes, timestamps, and more
Error messages: Clear feedback on failures

Batch Operations

Upload multiple items efficiently:

log_batch(): Upload many metrics at once
log_images_batch(): Upload multiple images in parallel
Thread pool execution: Fast parallel uploads
Progress tracking: See upload status

Query Capabilities

Programmatically access your data:

list_runs(): Get all runs in a project
get_run(): Get details about a specific run
get_artifacts(): List all images/files for a run
download_artifact(): Download files locally
get_metrics(): Retrieve all metrics for a run
compare_runs(): Compare metrics across runs
list_projects(): List all your projects

Flexible Logging

Log any data type with simple, intuitive methods:

Metrics: Scalar values, arrays, and time-series data
Images: Matplotlib figures, PIL images, numpy arrays
Files: CSV, HDF5, JSON, and any other file format
Metadata: Attach custom metadata to any logged artifact

Project Organization

Organize your experiments hierarchically:

Projects: Top-level containers for related experiments
Runs: Individual experiment instances within a project
Tasks: Group multiple runs together for parameter sweeps or comparisons
Config: Store simulation parameters and settings with each run

Performance Optimized

Built for production use:

Non-blocking: Logging doesn’t slow down your simulations
Efficient: Minimal overhead, optimized for high-frequency logging
Robust: Automatic error handling and retry logic
Scalable: Works with single processes or thousands of MPI workers

Use Cases

Simulation Tracking

Track photonic simulations with frameworks like Meep, Tidy3D, or custom solvers:


import optixlog
import meep as mp
 
# Use context manager (recommended)
with optixlog.run(
    "waveguide_simulation",
    config={"wavelength": 1.55, "resolution": 30}
) as client:
    # Run simulation
    sim = mp.Simulation(...)
    for step in range(100):
        sim.run(until=1)
        power = calculate_power(sim)
        client.log(step=step, power=power)
        
        # Log field snapshot with one line!
        if step % 10 == 0:
            field = sim.get_array(...)
            client.log_array_as_image("field_snapshot", field, cmap='hot')

Parameter Sweeps

Systematically explore parameter spaces:


wavelengths = [1.3, 1.4, 1.5, 1.6]
for wavelength in wavelengths:
    with optixlog.run(
        run_name=f"sweep_{wavelength}",
        config={"wavelength": wavelength}
    ) as client:
        # Run simulation and log results
        client.log(step=0, transmission=transmission, reflection=reflection)
        
        # Log spectrum plot in one line!
        client.log_plot("spectrum", frequencies, transmission, 
                        title=f"Transmission (λ={wavelength})")

Hardware Testing

Automate hardware test equipment and log results:


import optixlog
import pyvisa
 
client = optixlog.init(run_name="spectrum_analyzer_test")
 
# Connect to instrument
rm = pyvisa.ResourceManager()
instrument = rm.open_resource("GPIB0::1::INSTR")
 
# Take measurements
for frequency in range(1000, 2000, 10):
    instrument.write(f"FREQ {frequency}MHz")
    power = float(instrument.query("POW?"))
    client.log(step=frequency, power_dBm=power)

MPI Parallel Simulations

Run distributed simulations with automatic process coordination:


# Works automatically with MPI
mpirun -n 4 python simulation.py
 
# In your code:
client = optixlog.init(run_name="parallel_sim")
# Only master process logs, workers are automatically handled

Architecture Overview

The SDK consists of several key components:

Core Client (`OptixClient`)

The main client class that handles all communication with the OptixLog API:

Initialization: Sets up connection and creates runs
Logging methods: log(), log_image(), log_file()
MPI coordination: Automatic master/worker detection and handling
Error handling: Comprehensive error messages and recovery

Helper Functions

Utility functions for common operations:

optixlog.init(): Convenient initialization function
get_mpi_info(): Get current MPI environment information
is_master_process(): Check if current process is master
create_project(): Create new projects programmatically

MPI Detection

Automatic detection of MPI environments through multiple methods:

Environment variables (OMPI_COMM_WORLD_RANK, PMI_RANK, etc.)
mpi4py library (if available)
Meep’s MPI detection (if Meep is installed)

Python Version Requirements

Minimum: Python 3.8
Recommended: Python 3.9 or higher
Tested on: Python 3.8, 3.9, 3.10, 3.11, 3.12

Dependencies

Required

requests: HTTP client for API communication
numpy: Numerical operations
matplotlib: Plotting support
pillow: Image processing
rich: Colored console output (optional but recommended)

Optional

mpi4py: MPI support for parallel simulations
meep: Integration with Meep FDTD simulator