========================================= == Building new protocols, the scapy way == == Fred Raynal == == fred(at)security-labs.org == ========================================= This article explains how to build a new protocol within scapy. There are 2 main objectives: - Dissecting : this is done when a packet is received (from the network or a file) and should be converted to scapy's internals. - Building : When one wants to send such a new packet, some stuff needs to be adjusted automatically in it. =============================== = Not "packets" but "layers" = =============================== Before digging into dissection itself, let us look at how packets are organized. >>> p = IP()/TCP()/"AAAA" >>> p >> >>> p.summary() 'IP / TCP 127.0.0.1:ftp-data > 127.0.0.1:www S / Raw' We are interested in 2 "inside" fields of the class Packet: - p.underlayer - p.payload And here is the main "trick". You do not care about packets, only about layers, stacked one after the other. One can easily access a layer by its name : p[TCP] returns the TCP and followings layers. This is a shortcut for p.getlayer(TCP). Tip : there is an optional argument (nb) which returns the nb^th layer of required protocol Let's put everything together now, playing with the TCP layer: >>> tcp=p[TCP] >>> tcp.underlayer >> >>> tcp.payload As expected, tcp.underlayer points to the beginning of our IP packet, and tcp.payload to its payload. * Building a new layer ====================== VERY EASY ! A layer is mainly a list of fields. Let's look at UDP definition: class UDP(Packet): name = "UDP" fields_desc = [ ShortEnumField("sport", 53, UDP_SERVICES), ShortEnumField("dport", 53, UDP_SERVICES), ShortField("len", None), XShortField("chksum", None), ] And you are done ! There are many fields already defined for convenience, look at the doc^W sources as Phil would say. So, defining a layer is simply gathering fields in a list. The goal is here to provide the efficient default values for each field so the user does not have to give them when he builds a packet. The main mechanism is based on the Field structure. Always keep in mind that a layer is just a little more than a list of fields, but not much more. So, to understanding how layers are working, one needs to look quickly at how the fields are handled. * Manipulating packets == manipulating its fields ================================================= A field should be considered in different states: - i(nternal) : this is the way scapy manipulates it. - m(achine) : this is where the truth is, that is the layer as it is on the network. - h(uman) : how the packet is displayed to our human eyes. This explains the mysterious methods i2h(), i2m(), m2i() and so on available in each field: they are conversion from one state to another, adapted to a specific use. Other special functions: - any2i() guess the input representation and returns the internal one. - i2repr() a nicer i2h() However, all these are "low level" functions. The functions adding or extracting a field to the current layer are: - addfield(self, pkt, s, val): copy the network representation of field val (belonging to layer pkt) to the raw string packet s. class StrFixedLenField(StrField): def addfield(self, pkt, s, val): return s+struct.pack("%is"%self.length,self.i2m(pkt, val)) - getfield(self, pkt, s): extract from the raw packet s the field value belonging to layer pkt. It returns a list, the 1st element is the raw packet string after having removed the extracted field, the second one is the extracted field itself in internal representation class StrFixedLenField(StrField): def getfield(self, pkt, s): return s[self.length:], self.m2i(pkt,s[:self.length]) When defining your own layer, you usually just need to define some *2*() methods, and sometimes also the addfield() and getfield(). * Example : variable length quantities ====================================== There is way to represent integers on a variable length quantity often used in protocols, for instance when dealing with signal processing (e.g. MIDI). Each byte of the number is coded with the MSB set to 1, except the last byte. For instance, 0x123456 will be coded as 0xC8E856: def vlenq2str(l): s = [] s.append( hex(l & 0x7F) ) l = l >> 7 while l>0: s.append( hex(0x80 | (l & 0x7F) ) ) l = l >> 7 s.reverse() return "".join(map( lambda(x) : chr(int(x, 16)) , s)) def str2vlenq(s=""): i = l = 0 while i>> f = FOO(data="A"*129) >>> f.show() ###[ FOO ]### len= 0 data= 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' Here, len is not yet computed and only the default value are displayed. This is the current internal representation of our layer. Let's force the computation now: >>> f.show2() ###[ FOO ]### len= 129 data= 'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' The method show2() displays the fields with their values as they will be sent to the network, but in a human readable way, so we see len=129 Last but not least, let us look now at the machine representation: >>> str(f) '\x81\x01AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA' The first 2 bytes are \x81\x01, which is 129 in this encoding. ============== = Dissecting = ============== Layers are only list of fields, but what is the glue between each field, and after, between each layer. These are the mysteries explain in this section. * The basic stuff ================= The core function for dissection is Packet.dissect(): def dissect(self, s): s = self.pre_dissect(s) s = self.do_dissect(s) s = self.post_dissect(s) payl,pad = self.extract_padding(s) self.do_dissect_payload(payl) if pad and conf.padding: self.add_payload(Padding(pad)) When called, s is a string containing what is going to be dissected. self points to the current layer. >>> p=IP("A"*20)/TCP("B"*32) WARNING: bad dataofs (4). Assuming dataofs=5 >>> p >> Packet.dissect() is called 3 times: 1. to dissect the "A"*20 as an IPv4 header 2. to dissect the "B"*32 as a TCP header 3. and since there are still 12 bytes in the packet, they are dissected as "Raw" data (which is some kind of default layer type) For a given layer, everything is quite straightforward. - pre_dissect() is called to prepare the layer. - do_dissect() perform the real dissection of the layer. - post_dissection() is called when some updates are needed on the dissected inputs (e.g. deciphering, uncompressing, ... ) - extract_padding() is an important function which should be called by every layer containing its own size, so that it can tell apart in the payload what is really related to this layer and what will be considered as additional padding bytes. - do_dissect_payload() is the function in charge of dissecting the payload (if any). It is based on guess_payload_class() (see below). Once the type of the payload is known, the payload is bound to the current layer with this new type: def do_dissect_payload(self, s): cls = self.guess_payload_class(s) p = cls(s, _internal=1, _underlayer=self) self.add_payload(p) At the end, all the layers in the packet are dissected, and glued together with their known types. * Dissecting fields =================== The method with all the magic between a layer and its fields is do_dissect(). If you have understood the different representations of a layer, you should understand that "dissecting" a layer is building each of its fields from the machine to the internal representation. Guess what? That is exactly what do_dissect() does: def do_dissect(self, s): flist = self.fields_desc[:] flist.reverse() while s and flist: f = flist.pop() s,fval = f.getfield(self, s) self.fields[f] = fval return s So, it takes the raw string packet, and feed each field with it, as long as there are data or fields remaining >>> FOO("\xff\xff"+"B"*8) When writing FOO("\xff\xff"+"B"*8), it calls do_dissect(). The first field is VarLenQField. Thus, it takes bytes as long as their MSB is set, thus until (and including) the first 'B'. This mapping is done thanks to VarLenQField.getfield() and can be cross-checked: >>> vlenq2str(2097090) '\xff\xffB' Then, the next field is extracted the same way, until 2097090 bytes are put in FOO.data (or less if 2097090 bytes are not available, as here). If there are some bytes left after the dissection of the current layer, it is mapped in the same way to the what the next is expected to be (Raw by default): >>> FOO("\x05"+"B"*8) > Hence, we need now to understand how layers are bound together. * Binding layers ================ One of the cool features with scapy when dissecting layers is that is try to guess for us what the next layer is. The official way to link 2 layers is using bind_layers(): For instance, if you have a class HTTP, you may expect that all the packets coming from or going to port 80 will be decoded as such. This is simply done that way: bind_layers( TCP, HTTP, sport=80 ) bind_layers( TCP, HTTP, dport=80 ) That's all folks ! Now every packet related to port 80 will be associated to the layer HTTP, whether it is read from a pcap file or received from the network. * the guess_payload_class() way Sometimes, guessing the payload class is not as straightforward as defining a single port. For instance, it can depends on a value of a given byte in the current layer. The 2 needed methods are: - guess_payload_class() which must return the guessed class for the payload (next layer). By default, it uses links between classes that have been put in place by bind_layers(). - default_payload_class() which returns the default value. This method defined in the class Packet returns Raw, but it can be overloaded. For instance, decoding 802.11 changes depending on whether it is ciphered or not: class Dot11(Packet): def guess_payload_class(self, payload): if self.FCfield & 0x40: return Dot11WEP else: return Packet.guess_payload_class(self, payload) Several comments are needed here: - this cannot be done using bind_layers() because the tests are supposed to be "field==value", but it is more complicated here as we test a single bit in the value of a field. - if the test fails, no assumption is made, and we plug back to the default guessing mechanisms calling Packet.guess_payload_class() Most of the time, defining a method guess_payload_class() is not a necessity as the same result can be obtained from bind_layers(). * Changing the default behavior If you do not like scapy's behavior for a given layer, you can either change or disable it through the call to split_layer(). For instance, if you do not want UDP/53 to be bound with DNS, just add in your code: split_layers(UDP, DNS, sport=53) Now every packet with source port 53 will not be handled as DNS, but whatever you specify instead. * Under the hood : putting everything together ============================================== In fact, each layer has a field payload_guess. When you use the bind_layers() way, it adds the defined next layers to that list. >>> p=TCP() >>> p.payload_guess [({'dport': 2000}, ), ({'sport': 2000}, ), ... )] Then, when it needs to guess the next layer class, it calls the default method Packet.guess_payload_class(). This method runs through each element of the list payload_guess, each element being a tuple: - the 1st value is a field to test ('dport': 2000) - the 2nd value is the guessed class if it matches (Skinny) So, the default guess_payload_class() tries all element in the list, until one matches. If no element are found, it then calls default_payload_class(). If you have redefined this method, then yours is called, otherwise, the default one is called, and Raw type is returned. Packet.guess_payload_class() - test what is in field guess_payload - call overloaded guess_payload_class() ============ = Building = ============ Building a packet is as simple as building each layer. Then, some magic happens to glue everything. Let's do magic then. * The basic stuff ================= First thing to establish: what does "build" mean? As we have seen, a layer can be represented in different ways (human, internal, machine). Building means going to the machine format. Second thing to understand is _when_ a layer is built. Answer is not that obvious, but as soon as you need the machine representation, the layers are built: when the packet is dropped on the network or written to a file, when it is converted as a string, ... In fact, machine representation should be regarded as a big string with the layers appended altogether. >>> p = IP()/TCP() >>> hexdump(p) 0000 45 00 00 28 00 01 00 00 40 06 7C CD 7F 00 00 01 E..(....@.|..... 0010 7F 00 00 01 00 14 00 50 00 00 00 00 00 00 00 00 .......P........ 0020 50 02 20 00 91 7C 00 00 P. ..|.. Calling str() builds the packet: - non instanced fields are set to their default value - lengths are updated automatically - checksums are computed - and so on. In fact, using str() rather than show2() or any other method is not a random choice as all the functions building the packet calls Packet.__str__(). However, __str__() calls another method: build(): def __str__(self): return self.__iter__().next().build() What is important also to understand is that usually, you do not care about the machine representation, that is why the human and internal representations are here. So, the core method is build() (the code has been shortened to keep only the relevant parts): def build(self,internal=0): pkt = self.do_build() pay = self.build_payload() p = self.post_build(pkt,pay) if not internal: pkt = self while pkt.haslayer(Padding): pkt = pkt.getlayer(Padding) p += pkt.load pkt = pkt.payload return p So, it starts by building the current layer, then the payload, and post_build() is called to update some late evaluated fields (like checksums). Last, the padding is added to the end of the packet. Of course, building a layer is the same as building each of its fields, and that is exactly what do_build() does. * Building fields ================= The building of each field of a layer is called in Packet.do_build(): def do_build(self): p="" for f in self.fields_desc: p = f.addfield(self, p, self.getfieldval(f)) return p The core function to build a field is addfield(). It takes the internal view of the field and put it at the end of p. Usually, this method calls i2m() and returns something like p.self.i2m(val) (where val=self.getfieldval(f)). If val is set, then i2m() is just a matter of formatting the value the way it must be. For instance, if a byte is expected, truct.pack("B", val) is the right way to convert it. However, things are more complicated if val is not set, it means no default value was provided earlier, and thus the field needs to compute some "stuff" right now or later. "Right now" means thanks to i2m(), if all pieces of information is available. For instance, if you have to handle a length until a certain delimiter. Ex: counting the length until a delimiter class XNumberField(FieldLenField): def __init__(self, name, default, sep="\r\n"): FieldLenField.__init__(self, name, default, fld) self.sep = sep def i2m(self, pkt, x): x = FieldLenField.i2m(self, pkt, x) return "%02x" % x def m2i(self, pkt, x): return int(x, 16) def addfield(self, pkt, s, val): return s+self.i2m(pkt, val) def getfield(self, pkt, s): sep = s.find(self.sep) return s[sep:], self.m2i(pkt, s[:sep]) In this example, in i2m(), if x has already a value, it is converted to its hexadecimal value. If no value is given, a length of "0" is returned. The glue is provided by Packet.do_build() which calls Field.addfield() for each field in the layer, which in turn calls Field.i2m(): the layer is built IF a value was available. * Handling default values: post_build ===================================== A default value for a given field is sometimes either not known or impossible to compute when the fields are put together. For instance, if we used a XNumberField as defined previously in a layer, we expect it to be set to a given value when the packet is built. However, nothing is returned by i2m() if it is not set. The answer to this problem is Packet.post_build(). When this method is called, the packet is already built, but some fields still need to be computed. This is typically what is required to compute checksums or lengths. In fact, this is required each time a field's value depends on something which is not in the current So, let us assume we have a packet with a XNumberField, and have a look to its building process: class Foo(Packet): fields_desc = [ ByteField("type", 0), XNumberField("len", None, "\r\n"), StrFixedLenField("sep", "\r\n", 2) ] def post_build(self, p, pay): if self.len is None and pay: l = len(pay) p = p[:1] + hex(l)[2:]+ p[2:] return p+pay When post_build() is called, p is the current layer, pay the payload, that is what has already been built. We want our length to be the full length of the data put after the separator, so we add its computation in post_build(). >>> p = Foo()/("X"*32) >>> p.show2() ###[ Foo ]### type= 0 len= 32 sep= '\r\n' ###[ Raw ]### load= 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX' len is correctly computed now >>> hexdump(str(p)) 0000 00 32 30 0D 0A 58 58 58 58 58 58 58 58 58 58 58 .20..XXXXXXXXXXX 0010 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 58 XXXXXXXXXXXXXXXX 0020 58 58 58 58 58 XXXXX And the machine representation is the expected one. * Handling default values: automatic computation ================================================ As we have previously seen, the dissection mechanism is built upon the links between the layers created by the programmer. However, it can also be used during the building process. In the layer Foo(), our first byte is the type, which defines what comes next, e.g. if type=0, next layer is Bar0, if it is 1, next layer is Bar1, and so on. We would like then this field to be set automatically according to what comes next. class Bar1(Packet): fields_desc = [ IntField("val", 0), ] class Bar2(Packet): fields_desc = [ IPField("addr", "127.0.0.1") ] If we use these classes with nothing else, we will have trouble when dissecting the packets as nothing binds Foo layer with the multiple Bar*: >>> p = Foo()/Bar1(val=1337) >>> p > >>> p.show2() ###[ Foo ]### type= 0 len= 4 sep= '\r\n' ###[ Raw ]### load= '\x00\x00\x059' Problems: 1. type is still equal to 0 while we wanted it to be automatically set to 1. We could of course have built p with p = Foo(type=1)/Bar0(val=1337) but this is not very convenient. 2. the packet is badly dissected as Bar1 is regarded as Raw. This is because no links have been set between Foo() and Bar*(). As previously, we use bind_layers() to set everything correctly for us: bind_layers( Foo, Bar1, type=1 ) bind_layers( Foo, Bar2, type=2 ) Now, all the magic is there: >>> p = Foo()/Bar1(val=0x1337) >>> p > >>> p.show2() ###[ Foo ]### type= 1 len= 4 sep= '\r\n' ###[ Bar1 ]### val= 4919L Our 2 problems have been solved without us doing much: so good to be lazy :) * Under the hood : putting everything together ============================================== Last but not least, it is very useful to understand when each function is called when a packet is built: >>> hexdump(str(p)) Packet.str=Foo Packet.iter=Foo Packet.iter=Bar1 Packet.build=Foo Packet.build=Bar1 Packet.post_build=Bar1 Packet.post_build=Foo As you can see, it first runs through the list of each field, and then build them starting from the beginning. Once all layers have been built, it then calls post_build() starting from the end. =========== = History = =========== $Log: dissect.txt,v $ Revision 1.1 2007/01/31 10:58:21 raynal Initial revision