IST, black building, failure scenarios

    Integrated systems testing software for data centers where the whole facility has to fail on command and recover.

    Integrated systems testing — commonly Level 5 in a tiered commissioning programme — proves that power, cooling and controls work together under simulated failure, not just individually. Deskely holds every scenario, every system dependency and every observed result in one record set, so the black-building test produces evidence rather than a narrative report.

    Stage
    Level 5 / IST
    Scope
    Power, cooling, controls together
    Method
    Simulated failure scenarios
    Gate
    Facility acceptance
    FACILITY-WIDE — INTEGRATED SYSTEMS TESTIST MASTER SCRIPT · REV FIST CAMPAIGN — PHASE 2UTILITY LOSSIST-14UPS MODULE FAILIST-15CRAH FAILUREIST-16BMS FAILOVERIST-17RUNNING — FULL LOAD, ON UPSWITNESS SIGN-OFF PER SCENARIO — 18 WITNESSES22 SCENARIOS · 9 SYSTEMSTAG REGISTERFS-07Utility loss, train ARecords signedFS-12Chiller failure, N+1 proofIn testFS-18Generator start failure s…PreservationEPMS-A1Electrical power monitori…Punch open212 sequencesunder one system hierarchy

    The problem

    Every system passed alone; nobody has proven they pass together.

    By the time IST begins, generators, UPS, switchgear, chillers, CRAH units and building management controls have each been individually commissioned and signed off.

    Level 5 testing exists because individual system acceptance says nothing about what happens when a utility failure, a generator fault and a cooling loss are simulated in sequence against the live control logic.

    Deskely holds the scenario script, the expected sequence of operation and the actual observed result for every test, so a black-building test produces a comparable record rather than a witnessed anecdote.

    PASSED ALONE, NEVER PASSED TOGETHERLevel 5 — before the black-building testGeneratorsIndividually signedUPS and switchgearIndividually signedChillers and CRAHIndividually signedBuilding management controlsIndividually signedNothing shows what happens when utility loss, a generator fault and a cooling loss are simulated togetherSCENARIOS NOT YET RUN AGAINST LIVE CONTROL LOGICUtility loss + generator failure to startATS failure under loadChiller trip during simulated IT load0 of 6failure scenarios exercised end to end4 of 4individual systems already certifiedDeskely holds the scenario script and observed result as one comparable record.

    Setting up the register

    Scenarios, systems and sequences of operation are linked, not run from a separate script.

    Each IST scenario — utility loss, generator failure to start, ATS failure, chiller trip, CRAH failure, fire alarm interaction — is modelled as its own test record, linked to the systems and sequences of operation it exercises.

    The commissioning agent's scenario matrix and sequence of operation documents are parsed into the register, so the expected result for each step is recorded before the test, not reconstructed afterwards.

    Dependencies from earlier system-level commissioning — generator load bank results, ATS transfer times, cooling loop readiness — are visible against each scenario, because IST assumes they are already proven.

    1 · SOURCE DOCUMENTSIST master script rev F40 scenariosSequence of operations212 sequencesEPMS point list18 400 pointsLoad bank and load step …60 MW stagedPARSE + MATCHtag · description · drawing ref2 · ONE TAG REGISTERFS-07Utility loss, train ASCENARIOFS-12Chiller failure, N+1 proofSCENARIOFS-18Generator start failure s…SCENARIOEPMS-A1Electrical power monitori…CONTROLSTWO DOCUMENTS DISAGREEIST master script…Generator start failure sim…Sequence of opera…Abbreviated descriptionEngineer picks the sourcedecision loggedWeeks of transcription become a draft an engineer reviews.

    Execution

    Every step, every witness, captured in real time during the test.

    During the black-building test, witnesses from the owner, commissioning agent and contractors capture step timings, actual sequence of operation behaviour and observed deviations on tablets, offline where the network itself is part of the test.

    A deviation from the expected sequence — a slower-than-specified transfer, a controls interlock that did not fire — is raised as a finding immediately, tied to the exact scenario step and the system responsible.

    Findings are categorised by severity and by whether they require the scenario to be re-run, which is the decision that actually drives the IST schedule.

    SCENARIO MATRIX EXECUTIONLevel 5 — black-building testUtility loss, generator start and parallelingPassed, no deviationATS failure to transfer, unplanned lossPassed, no deviationChiller trip under simulated IT loadRunning nowFire alarm interaction with cooling and powerNot yet scheduledFacility acceptanceNot yet scheduled2 of 6scenarios completed and signed0facility acceptance before the matrix closesFacility acceptance is gated on every scenario in the matrix reaching a signed result.

    Facility acceptance

    Facility acceptance follows a completed scenario matrix, not a summary memo.

    Facility acceptance is gated on every scenario in the matrix reaching a signed result, with re-tests linked to the original finding so the history of what failed and what was fixed is not lost.

    The IST dossier — scenario scripts, sequence of operation records, timings and findings — becomes the reference the operations team uses for annual re-testing and for training staff on how the facility is meant to behave under failure.

    Findings that are accepted as residual risk rather than fixed stay visible with an owner, rather than disappearing once the certificate is issued.

    Scenario FS-07 — IST certification ladderEach scenario certificate cites the trend data and alarm log beneath itDRYScript review and dry runIssued 02 OctScript walkthrough signedcompleteRoles and radios confirmedconfirmedEXECScenario executionReady — full load scheduledFacility at test load58 MW / 60 MWAll upstream trains signedconfirmedSIGNPass/fail sign-offBlocked — awaiting executionTrend data reviewednot yet runDeviation reportnot yet run

    The black-building test is a multi-day negotiation, not one event

    Every stakeholder in the room has to sign the same result, and a failed scenario on day two can't quietly become a footnote by day four.

    A full IST for a large facility runs across several days and shifts, with the owner, the commissioning agent, the GC, the equipment vendors and often the future operations team all present as witnesses — and a scenario that fails at 2am on day two has to be captured with the same rigour as one witnessed by the owner's VP on day four.

    Handoffs between witnessing shifts are where findings traditionally get lost: a deviation noted verbally at shift change, written up loosely, and by the time the final acceptance meeting happens nobody can reconstruct exactly what the observed sequence of operation was or whether the re-test actually addressed the root cause.

    Deskely keeps every scenario's finding tied to the exact step, shift and witness who observed it, so a re-test scheduled for day four references the specific deviation from day two directly, and the final facility acceptance signature is made against a complete, continuous scenario history rather than a end-of-week summary reconstructed from notes.

    EVERY STEP, EVERY WITNESS, LIVEScenario 4 — ATS failure under loadCaptured in real time, offline where the network is itself part of the testSCENARIO SCRIPT — EXPECTEDATS transfers within 4 cycles, no interlock faultSequence of operation §3.23 witnesses recording — owner, CxA, contractorOBSERVED, TIMESTAMPEDUtility loss triggered, t+0.0sATS transfer initiated, t+0.08sInterlock fired late, t+0.31sDeviationFINDING — SCENARIO 4, STEP 3Interlock exceeded 4-cycle target — re-test requiredTied to ATS asset, categorised as re-test blockingFacility acceptance holds until the scenario re-runs clean.Deviations are raised against the exact scenario step, not reconstructed after a debrief.

    How it runs

    From individually proven systems to a facility proven under failure.

    IST is where a data centre commissioning programme is judged, because it is the only test that exercises the whole facility as the operator will actually experience it.

    1. 01

      Build the scenario matrix

      Utility loss, generator failure, cooling loss and controls interaction scenarios modelled as linked test records.

    2. 02

      Parse sequences of operation

      Expected system behaviour recorded against each scenario step before testing begins.

    3. 03

      Confirm system-level prerequisites

      Generator, ATS and cooling commissioning results checked as inputs before a scenario is scheduled.

    4. 04

      Run and witness the black building test

      Step timings and observed behaviour captured in real time by all witnessing parties on tablet.

    5. 05

      Raise and categorise findings

      Deviations tied to the scenario step and system, categorised by severity and re-test requirement.

    6. 06

      Certify facility acceptance

      Acceptance issued once the scenario matrix is complete, with the full dossier exported to operations.

    FAQ

    Questions about Integrated systems testing scopes.

    Ready to digitize
    commissioning and
    handover?

    We use cookies to improve your experience.
    You can opt out of certain cookies.
    Find out more in our privacy policy.