We leave the parameter tokens alone, considering it constant, while the parameter vec_out is used to hold the new stream of tokens. This allows the caller to have a before and after view on the token stream and reduces the worry of double frees.
We leave the parameter tokens alone, considering it constant, while the parameter vec_out is used to hold the new stream of tokens. This allows the caller to have a before and after view on the token stream and reduces the worry of double frees.