Deployment Tools

We have a set of development tools designed to deploy the EI code to a set of machines. This primarily was created for use on FABRIC, but the infrastructure can be applied to other node infrastructures as well. Please note that these tools are experimental and will likely require tuning to fulfill your desired deployment goals.

The repository is located at <REDACTED>. To use the tools, please install the following PIP packages:

pip install fabric fabrictestbed fabrictestbed-extensions

When using the FABRIC testbed extensions (FABlib), please ensure that you have set up your FABRIC credentials correctly (see this page).

Node Managers and EI Managers

The core class components of the deployment infrastructure are Node Managers and EI Managers. The former controls communication with nodes via SSH commands, and the latter controls EI processes on the nodes. EI Managers require a Node Manager when created.

Node Managers come in a few different forms, such as a local manager for the current machine, or a FABRIC manager for FABRIC nodes. The FABRIC node manager requires a JSON file containing node information, but local node managers do not.

An example of creating a local node manager with an EI manager:

TOPO_2NODE = '''
1.1 -- 1.2
'''

ndmgr = NodeManager('example')
node = Node("example-hostname")  # ssh hostname of local vm
node.dev_dir = "~/."
ndmgr.add_nodes([node])
ndmgr.asan = True
ei = EIManager(ndmgr)
ei.set_topo(TOPO_2NODE)

ndmgr.run('echo Node started').print()
ndmgr.wait_all(nodes={inst.node for int in ei.nodes})

This will create a local node manager that will use a 2-node single-domain topology when running EI processes, and also check that they are all reachable via running a simple echo command.

Using EI Managers

EI Managers are used to control services and run commands on nodes. Here’s a brief example:

SERVICES = ('snm', 'pt', 'hwqms')
for svc in SERVICES:
    getattr(ei, svc).start()
    time.sleep(0.1)
    res = getattr(ei, svc).status().wait()
    print(res)
ei.pt.config()

Note the last line that configures a service- some services require configuration once they are started, such as creating pipes and persistent CP connections between nodes.

To check if the host stack of the service is available, you can try to use seymour directly on a node:

for inst in ei.nodes:
    scripts_dir = f"{inst.node.dev_dir}/deploy/scripts"
    inst.node.run(
        f"cd {scripts_dir} && until python3 -c 'from seymour import HSSock ; HSSock(\"{inst.hssock}\")' >&/dev/null; do echo $(hostname): 'Waiting for host stack...' ; sleep 1 ; done"
    )