Controller "Unable to decode an event" error loop upon bad manifests

[jlewi]

I created a workflow using argo submit. The workflow is stuck in the pending state. Looking at the logs for the workflow controller I see errors

ERROR: logging before flag.Parse: E1228 21:43:36.059789       1 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: v1.Workflow: Spec: v1.WorkflowSpec: Templates: []v1.Template: v1.Template: Steps: [][]v1.WorkflowStep: []v1.WorkflowStep: ReadArrayCB: expect [ or n, but found: {, parsing 485 ..."steps":[{... at {"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"clusterName":"","creationTimestamp":"2017-12-28T21:16:42Z","deletionGracePeriodSeconds":null,"deletionTimestamp":null,"name":"e2e-test-1228-1311","namespace":"kubeflow-test-infra","resourceVersion":"130790","selfLink":"/apis/argoproj.io/v1alpha1/namespaces/kubeflow-test-infra/workflows/e2e-test-1228-1311","uid":"667379bf-ec14-11e7-b323-42010af0024d"},"spec":{"entrypoint":"e2e","templates":[{"name":"e2e","steps":[{"name":"checkout","template":"checkout"}]},{"container":{"command":["ls","-lR","/mnt/e2e-test-1228-1311"],"image":"busybox:latest","volumeMounts":[{"mountPath":"/mnt/e2e-test-1228-1311","name":"e2e-test-1228-1311"}]},"inputs":{"artifacts":[{"git":{"repo":"https://github.com/google/kubeflow.git","revision":"master"},"name":"kubeflow-source","path":"/mnt/e2e-test-1228-1311/kubeflow"},{"git":{"repo":"https://github.com/tensorflow/k8s.git","revision":"master"},"name":"tensorflow-k8s-source","path":"/mnt/e2e-test-1228-1311/tensorflow_k8s"}]},"name":"checkout"}],"volumeClaimTemplates":{"metadata":{"name":"e2e-test-1228-1311","namespace":"kubeflow-test-infra"},"spec":{"accessModes":["ReadWriteOnce"],"resources":{"requests":{"storage":"1Gi"}}}}}}

I'm guessing this is due to my spec being invalid somehow. Here's the manifest for my workflow:

---
apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: e2e-test-1228-1311
  namespace: kubeflow-test-infra
spec:
  entrypoint: e2e
  templates:
  - name: e2e
    steps:
    - - name: checkout
        template: checkout
  - container:
      command:
      - ls
      - -lR
      - /mnt/e2e-test-1228-1311
      image: busybox:latest
      volumeMounts:
      - mountPath: /mnt/e2e-test-1228-1311
        name: e2e-test-1228-1311
    inputs:
      artifacts:
      - git:
          repo: https://github.com/google/kubeflow.git
          revision: master
        name: kubeflow-source
        path: /mnt/e2e-test-1228-1311/kubeflow
      - git:
          repo: https://github.com/tensorflow/k8s.git
          revision: master
        name: tensorflow-k8s-source
        path: /mnt/e2e-test-1228-1311/tensorflow_k8s
    name: checkout
  volumeClaimTemplates:
  - metadata:
      name: e2e-test-1228-1311
      namespace: kubeflow-test-infra
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi

[jlewi]

Strange; I deleted the workflow and the workflow controller pod. I then resubmitted the workflow and it ran just fine.

[jlewi]

I haven't tried to reliably reproduce this but I seem to be observing the following

1. Submit an invalid workflow via kubectl

• e.g. set env on the container incorrectly

1. Workflow will be stuck in pending state and logs of workflow controller show "unable to decode"
2. Controller appears to be stuck in a failure mode; new workflows don't get processed; just keeps outputting "unable to decode"
3. UI also appears to be stuck; data doesn't show up.
4. Some combination of deleting the workflow controller pod and all workflows appears to fix things.

My expectation is that if I submit an invalid workflow spec via kubectl it will be marked as failed and not be stuck in the pending state.

If I submit the workflow via argo I get some validation. It would be great if submitting via kubectl also resulted in the spec being validated and any validation errors being reported in the workflow's status.

[jessesuen]

Unfortunately I've been aware of this issue for some time but haven't yet come across the proper solution. Basically the issue is that decode error is happening in the workflow informer, before any of the controller's resource handlers (AddFunc, UpdateFunc, DeleteFunc) even get a chance to handle the event and perform our own validation.

When the new, malformed workflow is created, the informer can't unmarshal the event into the expected datastructure (wfv1.Worfklow) typically because of mismatch data types. This is easily reproducible by running kubectl create against a workflow manifest where there is a mismatched data type for any fields. e.g.:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: decode-error-
spec:
  entrypoint: [whalesay]   # entrypoint is expected to be a string and not a list
  templates:
  - name: whalesay
    container:
      image: docker/whalesay:latest
      command: [cowsay]
      args: ["hello world"]

When we instantiate the informer, the constructor wants us to specify the runtime.Object (i.e. wfv1.Workflow) which the watch will decode to.

informer := cache.NewSharedIndexInformer(source, &wfv1.Workflow{}, workflowResyncPeriod, cache.Indexers{})

I need to do more research on how other controllers for CRDs are handling this. I would think that other CRD controllers would face the same issue when malformed CRs are created.

Another possibility of fixing this is through the JSONSchema/OpenAPI validation feature in 1.8. If K8s can enforce the schema early on, then it could guard against the event ever occurring.

[jessesuen]

BTW, until this is fixed, to save yourself some headache while developing the workflow YAMLs argo lint <wf.yaml> will be your friend (as well as argo submit)

[jessesuen]

FWIW, I looked at the sample-controller example from kubectl create a manifest which gives a list where the struct expects a string:

E1229 01:59:53.332796    3559 streamwatcher.go:109] Unable to decode an event from the watch stream: unable to decode watch event: v1alpha1.Foo: Spec: v1alpha1.FooSpec: DeploymentName: ReadString: expects " or n, but found [, error found in #10 byte of ...|entName":["example-f|..., bigger context ...|1e7-b3e3-4e0b1a127ace"},"spec":{"deploymentName":["example-foo"],"replicas":1}}|...

[jessesuen]

Filed . Hopefully a workaround will be provided since the controller ceases to function as soon as one bad manifest is created.

[jessesuen]

I'm happy to report that the workflow controller now guards against bad manifests by using a custom unstructured.Unstructured informer, then attempting to convert the unstructured object to a typed Workflow object. If this fails, the workflow is marked as failed with a "invalid spec" message.