fastsnmpy

Posted On Jan-17

Superfast parallel snmpbulkwalk

Fastsnmpy is a module that leverages python-extensions that come with net-snmp and provide highly parallelized faster-methods to walk oid-trees on devices.

In addition, it provides a method to bulkwalk mib-trees..

BulkWalk methods are missing from native python-bindings for net-snmp. By creating a wrapper around the GetBulk method instead, and maintaining state while traversing the oid-tree, fastsnmpy provides a clever solution to bulkwalk oids much faster

FastSNMPy provides the following:

  • snmpwalk(): Native python-bindings distributed with net-snmp, combined with fastsnmpy’s ability to parallelize snmpwalk operations.
  • snmpbulkwalk(): Ability to snmpbulkwalk devices, which makes it several magnitudes faster than net-snmp’s implementation of snmpwalk.
    • By leveraging the getbulk method, this module provides a quick snmpbulkwalk utility.
  • PROCESS-POOLS: By passing in a ‘workers=n’ attribute to the above methods, fastsnmpy can instantiate a process-pool to parallelize the snmpwalk and snmpbulkwalk methods, resulting in several devices being walked at the same time, effectively using all cores on a multicore machine.
  • One-Line, and Two-Line scripts that enable you to discover/walk all devices in a whole datacenter

Quick example – Running in interactive mode

Python 2.7.10 (default)

<blockquote>
  >> import netsnmp
  >> from fastsnmpy import SnmpSession
 
  >> hosts =['c7200-2','c7200-1','c2600-1','c2600-2']
  >> oids = ['ifDescr', 'ifIndex', 'ifName', 'ifDescr']
  >> newsession = SnmpSession ( targets = hosts, oidlist = oids, community='oznet' )
 
  >> results = newsession.snmpbulkwalk(workers=15)
  >> len(results)
  171031
  >>

Note: To use the module in scripts, please see the example.py included with the package.

Benchmarks

(1) Walking 30 nodes for ifDescr using snmpwalk():

time ./fastsnmpy.py
real    0m18.63s
user    0m1.07s
sys     0m0.38s

(2) Walking 30 nodes for ifDescr using snmpbulkwalk():

time ./fastsnmpy.py
real    0m9.17s
user    0m0.48s
sys     0m0.11s

(3) Walking 30 nodes for ifDescr using snmpwalk(workers=10):

time ./fastsnmpy.py
real    0m2.27s
user    0m2.87s
sys     0m0.66s

(4) Walking 30 nodes for ifDescr using snmpbulkwalk(workers=10):

time ./fastsnmpy.py
real    0m0.90s
user    0m2.44s
sys     0m0.40s

As you can see, fastsnmpy’s bulkwalk mode is almost 20 times faster than using python’s native snmp bindings for walking

Latest-version: Fastsnmpy2-1.2.1

  Download Here

Get from Git

Or fork from the git-repo GitHub-FastSNMPy

  • Rom

    Hello !

    I tested fastsnmp this morning. It’s a very nice module and very fast.
    But I’ve two problems :
    – in the “oids” param, I want to set an OID like .1.3.6.1.4.1.9.9.48.1.1.1.6.1 and not ifDescr.

    I tried it but the result is totally empty.
    And if I try with an “snmpwak -v2c -c COMMUNITY HOST .1.3.6.1.4.1.9.9.48.1.1.1.6.1” , it’s work.

    So I would to know how can I ask a host with an OID like this in your module please ?

    Thanks,
    Regards

    • Hey there. This shouldn’t cause a problem. fastsnmpy should understand both OIDs as well as MIBs.
      Here is an example with OIDs.

      >>> oids = [‘.1.3.6.1.2.1.1.1’]
      >>> hosts = [‘localhost’]
      >>> newsession = SnmpSession ( targets = hosts,oidlist = oids,community=’public’)
      >>> print newsession.snmpwalk(workers=1)
      Starting worker pool with 1 processes
      Getting .1.3.6.1.2.1.1.1 from localhost
      [
      {
      “hostname”: “localhost”,
      “val”: “Linux dev1 3.19.0-47-generic #53~14.04.1-Ubuntu SMP Mon Jan 18 16:09:14 UTC 2016 x86_64”,
      “oid”: “.1.3.6.1.2.1.1.1”,
      “iid”: “”,
      “tag”: “iso.3.6.1.2.1.1.1.0”,
      “type”: “OCTETSTR”
      }
      ]

      • Can you paste your run here (just like I did above), showing the empty variables that are returned?

        • Rom

          Hey !
          Yesterday I posted here for the lib https://github.com/kamakazikamikaze/fastsnmpy/blob/master/fastsnmpy.py. But it’s not exactly the same ^^. So first, which lib is better ?

          Your example worked for me.
          But I’ve always the problem with other OID.

          Here is my exemple :

          >>> if __name__ == ‘__main__’:

          >>> hosts =[‘myHost’]
          >>> oids = [‘.1.3.6.1.4.1.9.9.109.1.1.1.1.5.1′]

          >>> newsession = SnmpSession ( targets = hosts,
          >>> oidlist = oids,
          >>> community=’XXXX’)
          >>> print newsession.snmpwalk(workers=1)

          Starting worker pool with 1 processes
          Getting .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 from fwi-airbus-sca03
          []

          If I test on the ifDescr OID and the OID you gave me in your comment for the same host that’s work. But with this OID (.1.3.6.1.4.1.9.9.109.1.1.1.1.5.1) it’s not.
          For information it’s Cisco Firewall Oid to have CPU actual value.
          The snmpwalk bash command works on this oid & host.

          Thx to help me.

          • You should download the newer lib. It has a lot of bugfixes for similar issues.
            You can get it from https://pypi.python.org/pypi/fastsnmpy2/1.2.1
            Can you update to the newer lib, and see if you still have this problem.
            P.S – The newer update also addresses a memory-leak issue with the older lib.

          • Also, can you try the same mib, but just cut out the last 2 octets. ie use ‘.1.3.6.1.4.1.9.9.109.1.1.1.1’ instead of ‘.1.3.6.1.4.1.9.9.109.1.1.1.1.5.1’. Can you tell me if that makes a difference.

          • Rom

            Ok I updated to the newer lib and I still have this problem :

            Starting worker pool with 1 processes
            Getting .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 from fwi-airbus-sca03
            []

            If I cut the 2 last octets I’ve some informations but it’s just the next branchs in the Mib :

            Getting .1.3.6.1.4.1.9.9.109.1.1.1.1 from fwi-airbus-sca03
            [
            {
            “hostname”: “fwi-airbus-sca03”,
            “val”: “1”,
            “oid”: “.1.3.6.1.4.1.9.9.109.1.1.1.1”,
            “iid”: “”,
            “tag”: “enterprises.9.9.109.1.1.1.1.2.1”,
            “type”: “INTEGER”
            },
            {
            “hostname”: “fwi-airbus-sca03”,
            “val”: “1”,
            “oid”: “.1.3.6.1.4.1.9.9.109.1.1.1.1”,
            “iid”: “”,
            “tag”: “enterprises.9.9.109.1.1.1.1.3.1”,
            “type”: “GAUGE”
            },
            {
            “hostname”: “fwi-airbus-sca03”,
            “val”: “1”,
            “oid”: “.1.3.6.1.4.1.9.9.109.1.1.1.1”,
            “iid”: “”,
            “tag”: “enterprises.9.9.109.1.1.1.1.4.1”,
            “type”: “GAUGE”
            },
            {
            “hostname”: “fwi-airbus-sca03”,
            “val”: “1”,
            “oid”: “.1.3.6.1.4.1.9.9.109.1.1.1.1”,
            “iid”: “”,
            “tag”: “enterprises.9.9.109.1.1.1.1.5.1”,
            “type”: “GAUGE”
            }
            ]

            I don’t really understand why I can have these informations but not what I want on .1.3.6.1.4.1.9.9.109.1.1.1.1.5.1 …

          • Ok. That explains it.
            I’ll try to help you understand why this is happening.

            When you walk an SNMP tree, you are looking for all MIBS under that tree. The Linux netsnmp lib, which is used by fastsnmpy works this way. So for example, if you walk a tree x.y, you’ll see all its members UNDER it eg. x.y.1 , x.y.2 and so on.

            However if you directly try to walk x.y.1, there is nothing under it, so it won’t return anything. This is where the SNMPGET message comes in handy instead of SNMPWALK. Cuz you are going directly for the N-th element in the tree.

            Another way of looking at it , is that you are first asking the module to lookup all the houses in a zip code. Obviously this will work in any populated zip. But next you ask it to find all the houses inside a house – and it returns null.

            Since this is a limitation of the netsnmp module itself, my suggestion would be to query for the parent MIB I gave you, but then when you parse the response, you can just print the SNMP-OID from within the data structure. In other words, save the result into a variable, and do a json.reads on it . Then dump out only the tag inside each varbind.

            If you want an example , let me know. I can provide you some sample code I have which does that.

          • Rom

            OK ok I undertand now 🙂 thank you for the explanation. So I’ve to save the result into a variable and make a json.reads ont this variable ?

            Yes, I’m willing an example if you have it .
            Thanks,
            Regards

          • Sure. I’ll add it here when I get home this evening

          • Rom

            Perfect 🙂 Thank you again.

            If you have the time too, can you explain me the differences between snmpwalk() and snmpbulkwalk() ? I understand that snmpbulkwalk() is faster than snmpwalk()

            And more, what is the “worker” (I don’t see any difference if I change worker=5 to worker=10).
            Thank you for all
            Regards

          • Hey – so I implemented a fix in the code, and now I think it should give you what you need. Instead of stopping at the boundary of the root-tree, I am having it parse child trees.
            The updated version is on my github https://github.com/ajaysdesk/fastsnmpy/blob/master/dist/fastsnmpy2-1.2.2.tar.gz

            Try uninstalling the old version, and installing this new one. Also look at the example.py that comes with the package. Let me know if this works.

          • Romain

            Thank you a lot, that works :).
            Regards