Add PTF tests for basic forwarding exercise # 1#730
Add PTF tests for basic forwarding exercise # 1#730jafingerhut merged 7 commits intop4lang:masterfrom
Conversation
|
Hi @jafingerhut , this is a draft with the initial tests for the basic exercise (exercise 1). Before I add more tests (edge cases), I would appreciate any feedback on the overall approach. Also how you would like the testlib and veth_setup.sh dependencies be handled because they are external from the tutorial repo. I am happy to adjust. |
exercises/basic/runptf.sh
Outdated
| # Tests run against the solution P4 program. | ||
|
|
||
| # Path to p4runtime_shell_utils and testlib | ||
| T="`realpath ~/p4-guide/testlib`" |
There was a problem hiding this comment.
I cloned jafingerhut's p4-guide repo to hardcode the path here. This isn't in the tutorial repo. I would like some suggestions on how to handle this dependency.
There was a problem hiding this comment.
You should copy any files needed from the p4-guide repo into this one. I think you only use a couple of them. The license should be Apache 2 in them already, or if not, I can add that explicitly.
There was a problem hiding this comment.
Before you copy in the contents of the p4-guide/testutils directory, though, try deleting this line from the Python file:
import p4runtime_shell_utils as shu
I do not think your Python code uses the imported package at all, as far as I can tell, so you should not need it, which is good.
There was a problem hiding this comment.
I confirmed locally on my system with this PR's changes, but in addition my own local edits that deleted this line from the PTF test:
import p4runtime_shell_utils as shu
and removing these lines from runptf.sh:
T="`realpath ~/p4-guide/testlib`"
if [ x"${PYTHONPATH}" == "x" ]
then
P="${T}"
else
P="${T}:${PYTHONPATH}"
fi
and also later remove the command line options --pypath "$P" from the ptf command line, that the PTF test passes. I read through the code of the PTF test, and looked at the simple_switch log, and it looks like everything else is good there.
Please remove those things from this PR, as they are unnecessary.
There was a problem hiding this comment.
Hi @aqn96,
If I remember correctly, Andy developed these libraries for his repositories and for testing/playing with different setups. However, p4runtime_shell_utils does not exist in the tutorials repository. The equivalent there should be p4runtime_lib: https://github.com/p4lang/tutorials/tree/a36df0a61d7a60f1fbb3c212a46b828f8b1e4fc3/utils
There was a problem hiding this comment.
These examples show howp4runtime_libcan be used to configure a switch.
There was a problem hiding this comment.
Right. Except that for the moment, at least, the PTF test in this PR uses neither of those, and that is perfectly OK. It doesn't need anything provided by either of them.
There was a problem hiding this comment.
Hi @Dscano, thanks for the feedback! That makes sense. I'll refactor to use p4runtime_lib from the tutorials repo instead of the external testlib.
exercises/basic/README.md
Outdated
| ./runptf.sh | ||
| ``` | ||
|
|
||
| Note: The `veth` interfaces must be set up before running. If you see |
There was a problem hiding this comment.
One dependency is the veth interfaces set up which i ran the script from the repo commented. How should i handle this dependency for this tutorial repo?
There was a problem hiding this comment.
I think for now it would be best to create these new tests as things primarily for maintainers and contributors to this repository. I would prefer if we didn't mention them in the exercise README files yet that are intended for students using the tutorials.
There was a problem hiding this comment.
imho you can add a make test in the
Line 9 in a36df0a
run and then you extend it with the test.
There was a problem hiding this comment.
I also want to make sure I understand the make test suggestion correctly... are you thinking I should modify run_exercise.py to support a test mode like make a run_test.py where it reuses the environment setup from run_exercise.py (which handles interface creation) but runs PTF tests instead of spinning up Mininet?
There was a problem hiding this comment.
Sorry, I didn’t explain myself properly. What I meant is that you could add a make test target to the Makefile. This would allow users to run a test, observe how the program behaves, and integrate it into an automated testing pipeline/ Github Actions CI.
The make test target could include all the scripts and commands needed to run the test, such as creating the veth pairs, starting the switch, and so on.
In summary, it would be great to have a way to automate test execution for each exercise, so that we can integrate it into a CI/CD pipeline.
|
Once this is in we should create a Github action that runs this as part of CI. |
|
Updated based on @Dscano's feedback:
Still a draft as I plan to add maybe a couple more edge cases, but will adjust anything based on further feedback. @fruffy That would be a great next step. I'll look into setting up a GitHub Action once the test approach is finalized. |
|
@aqn96 overall looks good to me but I would like to test it locally, I'll do it asap. So if you want , in the meanwhile, you can aslo add the GitHub Action. |
Signed-off-by: aqn96 <aqnguyen96@gmail.com> Signed-off-by: An <aqnguyen96@gmail.com>
Signed-off-by: aqn96 <aqnguyen96@gmail.com> Signed-off-by: An <aqnguyen96@gmail.com>
Signed-off-by: aqn96 <aqnguyen96@gmail.com> Signed-off-by: An <aqnguyen96@gmail.com>
…ained veth setup, add make test target to the Makerfile, updated the README.md to reflect changes Signed-off-by: aqn96 <aqnguyen96@gmail.com> Signed-off-by: An <aqnguyen96@gmail.com>
Signed-off-by: An <aqnguyen96@gmail.com>
Signed-off-by: An <aqnguyen96@gmail.com>
|
Hi @Dscano @fruffy , I’ve updated this PR to include a fully automated CI pipeline for the basic forwarding exercise. Features of this CI setup:
Please feel free to give me feedback on this. |
…header Signed-off-by: An <aqnguyen96@gmail.com>
|
Just added 3 more edge cases to the current test suite:
I also updated the basic_fwd.py header to include the SPDX-License-Identifier and the community copyright notice as requested by @Dscano. I think this is correct as it matches other files in the repo and will keep that in mind moving forward. |
|
@jafingerhut any comments? |
|
Hi @aqn96, I have tested your code on VM P4 Tutorial Development 2025-04-01 , I got this: When you run |
jafingerhut
left a comment
There was a problem hiding this comment.
LGTM. I might want to factor out the veth_setup.sh and veth_teardown.sh into separate bash scripts later, but no need to do that before merging this.
exercises/basic/runptf.sh
Outdated
| # Tests run against the solution P4 program. | ||
|
|
||
| # Path to p4runtime_shell_utils and testlib | ||
| T="`realpath ~/p4-guide/testlib`" |
There was a problem hiding this comment.
You should copy any files needed from the p4-guide repo into this one. I think you only use a couple of them. The license should be Apache 2 in them already, or if not, I can add that explicitly.
exercises/basic/runptf.sh
Outdated
| # Tests run against the solution P4 program. | ||
|
|
||
| # Path to p4runtime_shell_utils and testlib | ||
| T="`realpath ~/p4-guide/testlib`" |
There was a problem hiding this comment.
Before you copy in the contents of the p4-guide/testutils directory, though, try deleting this line from the Python file:
import p4runtime_shell_utils as shu
I do not think your Python code uses the imported package at all, as far as I can tell, so you should not need it, which is good.
exercises/basic/runptf.sh
Outdated
| # Tests run against the solution P4 program. | ||
|
|
||
| # Path to p4runtime_shell_utils and testlib | ||
| T="`realpath ~/p4-guide/testlib`" |
There was a problem hiding this comment.
I confirmed locally on my system with this PR's changes, but in addition my own local edits that deleted this line from the PTF test:
import p4runtime_shell_utils as shu
and removing these lines from runptf.sh:
T="`realpath ~/p4-guide/testlib`"
if [ x"${PYTHONPATH}" == "x" ]
then
P="${T}"
else
P="${T}:${PYTHONPATH}"
fi
and also later remove the command line options --pypath "$P" from the ptf command line, that the PTF test passes. I read through the code of the PTF test, and looked at the simple_switch log, and it looks like everything else is good there.
Please remove those things from this PR, as they are unnecessary.
exercises/basic/README.md
Outdated
| ./runptf.sh | ||
| ``` | ||
|
|
||
| Note: The `veth` interfaces must be set up before running. If you see |
There was a problem hiding this comment.
I think for now it would be best to create these new tests as things primarily for maintainers and contributors to this repository. I would prefer if we didn't mention them in the exercise README files yet that are intended for students using the tutorials.
exercises/basic/runptf.sh
Outdated
| # Tests run against the solution P4 program. | ||
|
|
||
| # Path to p4runtime_shell_utils and testlib | ||
| T="`realpath ~/p4-guide/testlib`" |
There was a problem hiding this comment.
Right. Except that for the moment, at least, the PTF test in this PR uses neither of those, and that is perfectly OK. It doesn't need anything provided by either of them.


Addresses part of #103 issue (exercise 1 tests)
Adds automated PTF tests for the
basicexercise that run againstsolution/basic.p4.What's included:
ptf/basic_fwd.py— Three tests:DropTest— verifies packets are dropped with no table entries (default action)FwdTest— verifies single entry IPv4 forwarding with correct MAC rewrite and TTL decrementMultiEntryTest— verifies multiple LPM entries route to correct portsrunptf.sh— compiles the solution, starts simple_switch_grpc, runs PTF tests, cleans up.gitignore— excludes build artifacts and log filesSetup required:
vethinterfaces must be created before running tests. Useveth_setup.shfrom p4-guide or behavioral-modelrunptf.shcurrently references~/p4-guide/testlibforp4runtime_shell_utils.py. Open to suggestions on the best way to handle this dependency.Next steps:
Test results:
