Thrift virtual service architecture

A developer or tester uses a web browser to access the console. The console manages the virtual service. The system under test (application under test) connects directly to the virtual service on different ports.
Thrift architecture

Configuration

Thrift binary

Traffic Parrot needs access to the Thrift binary. The file name is thrift on Linux/Mac and thrift.exe on Windows.

To do this, simply copy your Thrift binary to the plugins directory of Traffic Parrot.

Traffic Parrot will also search the system path for the binary.

On startup you will see logs in trafficparrot.log informing that the binary was found, for example:

2020-03-06 15:59:27,611 INFO  main Thrift support enabled since there is a Java compiler and Thrift binary available.

You may see error logs in trafficparrot.log if the thrift binary is not found:

2020-03-06 15:59:27,611 WARN Thrift support disabled since Thrift compiler not available. Could not find 'thrift' binary in location(s): [/home/trafficparrot/plugins, /usr/local/bin]

Thrift files

Traffic Parrot needs to know about the schema of your messages. Provide your Thrift files that contain the API definitions.

To do this, simply copy your .thrift files into the trafficparrot-x.y.z/thrift directory and Traffic Parrot will use these definitions during record/replay.

On startup you will see logs in trafficparrot.log displaying which services were found and compiled, for example:

2020-03-06 15:59:27,611 INFO  main Thrift support enabled since there is a Java compiler and Thrift binary available.
----------------------------------------------------------------------
2020-03-06 15:59:28,343 INFO  main Compiling Thrift files in directory '/home/trafficparrot/./thrift' to temporary Java directory '/tmp/trafficparrot-thrift-java4691688603859252444' using binary '/home/trafficparrot/plugins/thrift': [shared.thrift, tutorial.thrift]
----------------------------------------------------------------------
2020-03-06 15:59:28,351 INFO  main Compiled 2 Thrift files to Java successfully
----------------------------------------------------------------------
2020-03-06 15:59:28,353 INFO  main Compiling Java classes from sources /tmp/trafficparrot-thrift-java4691688603859252444: [com.trafficparrot.example.InvalidOperation, com.trafficparrot.example.SharedService, com.trafficparrot.example.Calculator, com.trafficparrot.example.Work, com.trafficparrot.example.Operation, com.trafficparrot.example.SharedStruct]
----------------------------------------------------------------------
2020-03-06 15:59:29,656 INFO  main Compiled 6 Java classes successfully

You may see error logs in trafficparrot.log if the JDK or thrift binary is not found:

2020-03-06 15:59:27,611 WARN Thrift support disabled since Java compiler not available. Please use a JDK not a JRE to enable Thrift support.
----------------------------------------------------------------------
2020-03-06 15:59:27,611 WARN Thrift support disabled since Thrift compiler not available. Could not find 'thrift' binary in location(s): [/home/trafficparrot/plugins, /usr/local/bin]

SSL/TLS

Typically, Thrift is secured using SSL/TLS

Traffic Parrot has the following properties that can be used to configure SSL/TLS:
  • trafficparrot.virtualservice.thrift.tls.port
    The port number to use when recording/replaying TLS traffic
  • trafficparrot.virtualservice.thrift.server.trust.pem.url
    X.509 certificate collection file in PEM format that will be trusted by the virtual Thrift server
  • trafficparrot.virtualservice.thrift.server.pem.url
    X.509 certificate chain file in PEM format that will be the public key of the virtual Thrift server
  • trafficparrot.virtualservice.thrift.server.key.url
    PKCS#8 private key file in PEM format that will be the private key of the virtual Thrift server
  • trafficparrot.virtualservice.thrift.client.trust.pem.url
    X.509 certificate collection file in PEM format that will be trusted by the virtual Thrift client
If client authentication is required use the following additional properties:
  • trafficparrot.virtualservice.thrift.client.pem.url
    X.509 certificate chain file in PEM format that will be the public key of the virtual Thrift server
  • trafficparrot.virtualservice.thrift.client.key.url
    PKCS#8 private key file in PEM format that will be the private key of the virtual Thrift server

The URL property values can be set to classpath entries e.g. classpath:certificates/thrift/server.pem

By default Traffic Parrot has properties that correspond to the certificates found in the trafficparrot-x.y.z/certificates/thrift directory.

Alternatively, you can specify file URL entries e.g. file:///path/to/server.pem

The property trafficparrot.virtualservice.thrift.non.tls.port specifies a port that can be used if SSL/TLS is not required.

Performance

The following properties have a significant impact on Thrift performance during replay:

Property Description
trafficparrot.virtualservice.mapping.cache.milliseconds Mapping files can be cached in memory during replay to improve performance. The default value of 0 means do not cache at all.
trafficparrot.virtualservice.mapping.cache.populate.on.startup When true the mapping file cache will be populated immediately on startup.

Recording and replaying Thrift

Introduction

During recording, incoming and outgoing messages are matched up to provide mappings. Then upon playback receipt of a matching incoming message will trigger generation of an outgoing message.

This diagram shows how two production systems connect:
Thrift communication

One system sends a Thrift request message to another system, which consumes the messages and returns a Thrift response message to the first system. If our goal is to test the system-under-test in isolation, we must record these interactions in order to replay them.

Recording and replaying Thrift messages

As you can see in the diagram below, the virtual service acts as a proxy between the Thrift client and Thrift server. The provided Thrift files are used to determine the message schema of the request and response messages for the requested Thrift method. Mappings are saved representing request/response pairs that have been recorded.

Recording Thrift messages using provided Thrift files
Steps to perform a recording using provided Thrift files
  1. Start Traffic Parrot
  2. Configure Traffic Parrot with your Thrift files
  3. Go to the Thrift record page
  4. Enter the host and port of the Thrift server to record
  5. Click "Start recording"
  6. Reconfigure your system under test to connect to the virtual service Thrift server (by default this is localhost:5562)
  7. Execute a test case to allow Traffic Parrot to record requests and responses. They will appear on the Messages and Mappings lists on the web page as shown on Mappings and messages lists during recording image.
  8. Click "Stop recording"

As the system under test generates messages and receives responses they are paired into mappings and listed in the 'Mappings' table.

Mappings list during a Thrift recording

On replay, the provided proto files are again used to determine the schema of the requested Thrift method. The request is matched against the mappings to find the corresponding response message.

Replaying Thrift messages using provided proto files
Steps to perform a replay using provided Thrift files
  1. Start Traffic Parrot
  2. Configure Traffic Parrot with your Thrift files
  3. Reconfigure your system under test to connect to the virtual service Thrift server (by default this is localhost:5562)
  4. Traffic Parrot will replay recorded mappings when it is not in record mode

Existing mappings

If there are existing mappings present before a recording starts, these mappings will not be used to return responses instead of recording a new response.

For example, if there is a mapping for method Calculator/calculate then any traffic to Calculator/calculate will record a new response and the existing mapping will not be considered.

Please contact support@trafficparrot.com if you have a requirement for a mixed recording mode similar to HTTP recording where existing mappings during recording are used to provide responses instead of recording a new response.

Add/Edit Thrift mappings

Usage

Thrift messages as JSON

Traffic Parrot uses JSON to represent Thrift message payloads. It stores the Thrift request and response message payloads as JSON.

This means, the request and response body can be edited as if they were a JSON body. You are free to change the JSON as you wish, so long as the edits are compatible with the underlying proto message schema that the JSON represents.

Editing a mapping

First, go to Thrift in the top navigation bar and click Add/Edit.

Add Thrift mapping screenshot

Fill in the Request/Response fields and click Save to configure a mapping.

After saving the mapping, it will appear in the list of mappings.

Thrift mapping list

Clicking the edit button will allow you to edit an existing mapping.

Supported features
  • Edit response message values
  • Edit response message structure (e.g. delete optional fields, add items to list, add nested fields)
  • Edit exception message
  • Use helpers in the response body
  • Use parts of the request message in the response message by extracting with JsonPath
  • Match request messages using the request body matchers (e.g. JsonPath, contains, regexp) on the JSON representation of the message
  • Use priority to set up a preference order for matching mappings
Currently, editing non-exception metadata is not yet possible. Please contact support@trafficparrot.com to be notified of when this functionality is ready.

Priority

The request priority can be set in order to set up a preference order for matching mappings.

The highest priority value is 1. If two or more mappings both match a request, the mapping with the higher priority will be used to provide the response. The default priority is 5.

This can be useful, if you want a "catch-all" mapping that returns a general response for most requests and specific mappings on top that return more specific responses.

Thrift request matchers

When Traffic Parrot receives a request, it will try to simulate the system it is replacing by sending back a response to the client that sent the request. To decide which response to send, it will go through all the request to response mappings it has available to find the response to be returned. For more details how request matching works, see Request matching.

There are several matchers available to match Thrift requests, depending on the attribute.

Thrift method
Will match only the selected Thrift method name. For example:
Calculator/calculate
Request message matchers

Traffic Parrot uses JSON to represent Thrift Protocol Buffers message payloads.

For example, if you define a service:
enum Operation {
  ADD = 1,
  SUBTRACT = 2,
  MULTIPLY = 3,
  DIVIDE = 4
}

struct Work {
  1: i32 num1 = 0,
  2: i32 num2,
  3: Operation op,
  4: optional string comment,
}

exception InvalidOperation {
  1: i32 whatOp,
  2: string why
}

service Calculator extends shared.SharedService {
   i32 calculate(1:Work w) throws (1:InvalidOperation ouch)
}
then, if you choose "equal to JSON representation" as the request body matcher and populate the request body with:
{
  "w" : {
    "num1" : 1,
    "num2" : 9,
    "op" : "ADD",
    "comment" : null
  }
}
then Traffic Parrot will match this request any time it receives a Thrift request message for the operation 1 ADD 9

You can use not only the "equal to JSON representation" matcher but any of the ones described below.

Matcher name Matcher Id Description
any any Any Thrift request payload will match.
equal to JSON representation equalToJson Check that the received request payload has all attributes equal to the ones specified in the mapping as JSON.
matches JSON representation matchesJson

Check that the received request payload matches (allowing for special wildcard tokens) attributes specified in the mapping as JSON.

Tokens allowed:
  • {{ anyValue }} - matches any value
  • {{ anyNumber }} - matches any whole or decimal number
  • {{ anyElements }} - matches any number of sub-elements (child-nodes)
For example a "matches JSON" request body matcher:
{
  "name": "{{ anyValue }}",
  "lastName": "{{ anyValue }}",
  "age": "{{ anyNumber }}",
  "children": "{{ anyElements }}"
}
will match a request payload JSON representation:
{
  "name": "Bob",
  "lastName": "Smith",
  "age": 37,
  "children": [{"name": "sam"}, {"name": "mary"}]
}
matches JSONPath matchesJsonPath Check that the received request payload, when converted by Traffic Parrot to JSON, matches JSONPath specified in the mapping. For example, given a Thrift message:
struct Work {
  1: i32 num1 = 0,
  2: i32 num2,
  3: Operation op,
  4: optional string comment,
}
when Traffic Parrot receives a "Work" request object it will convert it to JSON for matching purposes. A sample "Work" message, when received by Traffic Parrot and converted to JSON would look like this:
{
  "w" : {
    "num1" : 1,
    "num2" : 9,
    "op" : "ADD",
    "comment" : null
  }
}
And then, a sample "matches JSONPath" matcher to match the message above could look like this:
$[?(@.op == 'ADD' && @.num1 == 1 && @.num2 == 9)]
contains contains Check that the received request payload, when converted to JSON, contains the sequence of characters specified in the mapping
matches regex matches Check that the received request payload, when converted to JSON, matches the regexp specified in the mapping. For example, given a proto message:
struct Work {
  1: i32 num1 = 0,
  2: i32 num2,
  3: Operation op,
  4: optional string comment,
}
when Traffic Parrot receives a "Work" message it will convert it to JSON for matching purposes. A sample "Work" message, when received by Traffic Parrot and converted to JSON would look like this:
{
  "w" : {
    "num1" : 1,
    "num2" : 9,
    "op" : "ADD",
    "comment" : null
  }
}
And then, a sample "matches regex" matcher to match the message above could look like this:
.*"num1" : 1.*"num2" : 9.*"op" : "ADD".*
does not match regexp doesNotMatch Check that the received request payload, when converted to JSON, does not match the regexp specified in the mapping. See "matches regexp" above for more details.

Proof of concept with on-premises installation at a large enterprise

See proof of concept in the user guide.

Dynamic responses and custom extensions